The Ultimate Facial-Recognition Algorithm

Human memory is notoriously unreliable. Even people with the sharpest facial-recognition skills can only remember so much.

It’s tough to quantify how good a person is at remembering. No one really knows how many different faces someone can recall, for example, but various estimates tend to hover in the thousands—based on the number of acquaintances a person might have.

Machines aren’t limited this way. Give the right computer a massive database of faces, and it can process what it sees—then recognize a face it’s told to find—with remarkable speed and precision. This skill is what underpins the enormous promise of facial-recognition software in the 21st century. It’s also what makes contemporary surveillance systems so creepy.

The thing is, machines still have limitations when it comes to facial recognition. And scientists are only just beginning to understand what those constraints are. To begin to figure out how computers are struggling, researchers at the University of Washington created a massive database of faces—they call it MegaFace—and tested a variety of facial recognition algorithms as they scaled up in complexity. The idea was to test the machines on a database that included up to 1 million different images of nearly 700,000 different people—and not just a large database featuring a relatively small number of different faces, more consistent with what’s been used in other research.

As the databases grew, machine accuracy dipped across the board. Algorithms that were right 95 percent of the time when they were dealing with a 13,000-image database, for example, were accurate about 70 percent of the time when confronted with 1 million images. That’s still pretty good, says one of the researchers, Ira Kemelmacher-Shlizerman. “Much better than we expected,” she told me.

But she and her colleagues believe accuracy would continue to decline as gallery sizes grow, which poses a real problem for the kinds of applications people want and expect from facial-recognition software. “Many applications require accurate identification at planetary scale, finding the best matching face in a database of billions of people,” the researchers wrote in a paper announcing their findings. “This is truly like finding a needle in a haystack.”

Machines also had difficulty adjusting for people who look a lot alike—either doppelgängers, whom the machine would have trouble identifying as two separate people; or the same person who appeared in different photos at different ages or in different lighting, whom the machine would incorrectly view as separate people.

“Once we scale up, algorithms must be sensitive to tiny changes in identities and at the same time invariant to lighting, pose, age,” Kemelmacher-Shlizerman said.

The trouble is, for many of the researchers who’d like to design systems to address these challenges, massive datasets for experimentation just don’t exist—at least, not in formats that are accessible to academic researchers. Training sets like the ones Google and Facebook have are private. There are no public databases that contain millions of faces. MegaFace’s creators say it is the largest publicly available facial-recognition dataset out there.

“An ultimate face recognition algorithm should perform with billions of people in a dataset,” the researchers wrote.