scorecardresearch Skip to main content

Facial-recognition software precision may be years away

Low resolutions, tilted heads still confound

Tamerlan (left) and Dzhokhar Tsarnaev shown in security camera images on Boylston Street.FBI Photos

It looks simple in the movies. The hero feeds a digital image of a suspect into a computer, which matches it with millions of data files and comes up with a positive identification.

But in real life, it took eyeballs, not microchips, to identify Dzhokhar and Tamerlan Tsarnaev, the two young men believed to have bombed the Boston Marathon. Despite pictures of the brothers being in government databases, such as for motor vehicle licenses, the facial-recognition software was not able to match them against the fuzzy, grainy images investigators had of the suspects at the scene.

And it may be years before the technology works as well in real life as it does in the movies.


“That’s an open question that people who are developing systems are trying to solve,” said P. Jonathon Phillips, an electrical engineer at the National Institute of Standards and Technology who evaluates facial-recognition systems for the federal government. “We still do not know how long it will take.”

But even today, facial recognition can work well. The technology is commonly used by millions of amateurs who edit photos­ in programs such as Google Inc.’s Picasa. The software allows users to attach a name to a face in a photograph stored on the computer’s hard drive. Immediately, Picasa automatically locates the same face in every other photo on the computer.

Facial recognition has also proved successful in criminal investigations. Jennifer Lynch, an attorney for the Electronic Frontier Foundation, a high-tech civil liberties organization, estimates that about 35 states use facial-recognition technology to scan their databases of driver’s license photos for identity fraud.

In New Jersey, for example, the Motor Vehicle Commission ran facial-recognition software against its database of 19 million drivers’ photos, looking for duplicate images. The state found 669 cases that appeared to be deliberate examples of identity theft.


In February, the New Jersey attorney general’s office announced 38 arrests related to the investigation. Facial-recognition systems have also led to 2,500 identity theft arrests in New York State since 2010.

Using driver’s license photos is usually easy work for facial-recognition software because those images are of high quality. The subject is illuminated with bright, even light, looks directly into the camera, and avoids exaggerated facial expressions.

That makes it easy for the software to, for example, accurately measure the distance between the eyes, the width of the mouth, the texture of the skin, and the firmness of the underlying muscles.

”If I have a controlled driver’s license or mugshot photo and I want to match it to another photo, the performance can be as high as 99 percent,” said Anil Jain, a professor of computer science at Michigan State University and a specialist in facial recognition.

But the photos of the alleged Boston bombers, shot by low-resolution video surveillance cameras, were blurry and badly illuminated.

In some, the subjects are seen in profile, with one eye obscured. Jain said this angle makes it far harder for the software, which uses the distance between the eyes as a key factor in facial identification.

“As soon as you start changing the illumination, the pose, and expression, the accuracy starts dropping,” Jain said.

MorphoTrust USA in Billerica makes facial-recognition software that is licensed by the Federal Bureau of Investigation, though the company cannot say whether its products were used in the Boston investigation.


The company’s senior vice president for government operations, Jim Albers, said that while MorphoTrust’s facial-recognition software is “about as good as it’s going to get,” it is hampered by poor picture quality and off-center facial images.

Still, Albers said, as older surveillance cameras are gradually replaced by high-definition units, he believes facial recognition will become more successful.

Law enforcement agencies are not the only users of the technology.

The fire safety and security products giant Tyco International Ltd. uses facial recognition to track shoppers at retail stores as they move around, to detect shoplifting.

And an Italian company called Almax SpA makes a mannequin with a camera embedded in one eye. The mannequin captures images of nearby shoppers, classifying them by race, sex, and estimated age.

Neither product tries to identify individual shoppers, and an industry trade group, the Global Association For Marketing At Retail, has warned its members to protect customer privacy.

Michigan State’s Jain noted that smart criminals will be able to beat facial-recognition systems by wearing disguises, such as fake beards.

That means facial surveillance may prove more effective in tracking law-abiding citizens than terrorists.

Civil libertarians are already uneasy about the broad use of facial-recognition technology, fearing that it will be used to invade people’s privacy.

Courts have ruled that people have no right to privacy when walking down a public street. So police could someday use facial-recognition software and ever-present surveillance cameras to track any citizen’s movements — even, critics fear, those who are not suspected of wrongdoing.


“Once they prove it works, it’ll be all over the place,” predicted John Whitehead, president of the Rutherford Institute, a civil liberties organization in Rutherford, Va.

Jennifer Lynch of the Electronic Frontier Foundation said that the government should adopt restrictions on the use of the technology before it improves, such as to limit random surveillance.

Hiawatha Bray can be reached at