DHS Works to Improve Biometric Scanning of Masked Faces

Perhaps the most-worn accessories of 2020, face masks mark an unexpected new constant in people’s lives providing necessary protection against COVID-19—but they’re also known to pose some trouble for contemporary facial recognition systems. 

The Homeland Security Department, one of the government’s biggest biometrics systems users, is now steering research to confront the complexities limiting existing technology and help push forward tools to safely verify people’s identities at security checkpoints in a pandemic.    

Initial results from one recent effort “are actually quite promising,” according to Arun Vemury, director of DHS’ Biometric and Identity Technology Center. 

“We're getting to the point with this technology, where at least from the preliminary results, it looks like there's some combinations of biometric acquisition systems, the camera systems and the matching algorithms—when you combine them together, you could match eight or nine out of 10 people without asking them to remove their masks,” Vemury told Nextgov during a recent interview. “This means that for the vast majority of people in airports, they might not have to remove their masks anymore to even go through the security checks, and we could do a really good job of still matching them. So, I think it's very promising from that perspective. Is it 100%? Is it perfect? No. But it reduces the number of people who potentially have to take their masks off.”

Those outcomes Vemury pointed to emerged from the agency’s third annual Biometric Technology Rally, which was held over three weeks in late September and early October—and they also build off of a recent report led by the National Institutes of Science and Technology. The study was released in July as part of a planned series NIST’s Face Recognition Vendor Test, or FRVT, program is completing to assess the performance of face recognition algorithms on faces partially hidden by masks. For it, researchers reviewed almost 100 facial recognition systems and found that, at least for now, masks generally disrupt the tech’s accuracy.

Before COVID-19 hit, Vemury and his team were originally planning their next biometric rally, which they’d decided would evaluate biometrics systems’ ability to process people in small groups, to see how they might perform on families or with others, as many naturally travel. After the pandemic surfaced they shifted to plans for remote operations—and this more timely topic. 

Biometrics researchers within DHS and NIST have a “really great” and long-time-running relationship, Vemury noted, and had collaborated on the latter’s summer FRVT study.

 “[NIST’s] test was really a first order test,” Vemury explained. “It's really inexpensive and quick to run because it can all be done with computers.”

Using not Photoshop but a tool like it, NIST created digital masks over faces in DHS operational images. Officials subsequently compared how facial recognition algorithms performed on those photos, as well as images of the same people without synthetic masks edited on to cover their faces. It’s considered what Vemury called a “software simulated test.” 

“When the results there started looking really interesting was when we noticed that some of the matching algorithms didn't fail—they didn't fall apart. They only had small degradations in performance. So our goal was to take it to the next level, which was, when you look at a facial recognition system, it's not just a piece of software,” he said. “There are cameras or people interacting with the cameras, there are real face masks, and then there's how the system basically provides the result out. We do these larger scenario tests to understand how well certain products could actually be integrated into an overall capability.”

With sights set on advancing the original NIST-headed research and gaining a better grasp of potential performance in real-world scenarios, DHS turned its biometric rally into an opportunity to evaluate complete systems with real cameras, real people, real masks and facial recognition algorithms. Those algorithms and acquisition systems were offered up by industry. 

“Because NIST’s test is like a computer test, they can test a lot of stuff really quickly,” Vemury said. “We don't have a building big enough to hold all of the systems that people would want us to test. So, we have to be more selective.”

DHS selected six systems to test this go-around, which is fewer than usual for such events but made sense to abide by COVID-19 protocols. Conducted at the Maryland Test Facility, the latest rally put those face and/or iris acquisition systems to work, as well as 13 matching algorithms—with support from 582 human volunteers.

“There was a lot of Plexiglas involved and we ended up putting blue dots all over the floor where people could stand and sit to make sure we practiced social distancing,” Vemury said. “So, it was a unique experience. I never thought we were going to have to do something like that before, but I was really proud of the team about how we pulled together.”

On the ground, the systems were judged based on their ability to reliably snap photos of each volunteer with and without masks, as well as volunteers’ processing time and overall satisfaction. In the aftermath of the rally, DHS is shifting focus to assess the accuracy in the matching algorithms’ performance identifying each of the volunteers regardless of whether their faces are covered, using images from each of the systems being reviewed.   

With the heaps of photos collected, DHS is able to essentially standardize image output and get a strong grip on what all the cameras are capturing. Expanding on the preliminary outcomes, Vemury noted the systems are able to hone in on unique human features such as head shape, and faces’ periocular region—or the areas around individuals’ eyes—that remain uncovered when masks are worn.

“We're seeing some quite promising results regardless of the type of mask that's being used. Obviously, some masks seem to work better than others. But it's quite promising just that the information in this periocular region can provide so much value,” Vemury said. “One of the things I was concerned about was whether or not, if you do well at matching people with face masks, do you actually do worse at matching people without face masks? And surprisingly, for me, that actually wasn't the case.” 

Despite the encouraging results from the third rally, there isn’t a present plan to push out solutions evaluated into the department’s broadly used biometrics ecosystem.

“I think ideally, these things could be deployed quickly. It comes down to the flexibility of how quick the contracting process could work, perhaps,” Vemury said. “What we wanted to do though—because we're seeing such promising stuff out here—was to get it shared, and at least start getting some people aware of some of the possible benefits.”

DHS aims to release full—but thoughtfully curated—results of the effort by the end of the year. Vemury is also hoping and planning for another rally in 2021, though the focus remains unclear for now as it’s meant to reflect near-term needs in the biometric verification space.

“So, we may continue to look at face masks,” Vemury noted. “Or if things kind of get much better, we'll go back to looking at processing groups more efficiently, in a more intuitive way.”