Ideas

brainiac

How to automatically detect the most important people in a photograph

In the online, big data world, it’s important to be able to separate the wheat from the chaff. This is true when it comes to refining search results and culling a Twitter feed, and it’s true with photographs, too. When you search for pictures of a friend, you’re probably less interested in images of that person in the crowd at someone else’s wedding, and more interested in pictures in which they’re the main event.

Photo sites don’t know how to make that kind of distinction, yet, but a new innovation out of Virginia Tech and recently posted to arXiv.org might change that. It’s an algorithm (of course) that takes advantage of all sorts of social and technological cues to figure out who really matters in an image.

Advertisement

“We have the ability to look at a scene and, just by coding what people are doing, how people are looking at each other, we can get a sense of the important actors,” says Dhruv Batra, a professor of electrical and computer engineering and creator of the program, along with graduate student and lead designer Clint Solomon Mathialagan and Andrew Gallagher, an engineer at Google.

The program looks at 45 aspects of an image in order to rank the people in the image in terms of importance. These include some obvious factors. A person in the center of an image ranks higher, as does a person whose face is in focus, while a person whose face is partially obstructed ranks lower. They also include more subtle factors, like the directions in which people are looking. “If you look at a scene you can tell there is an important person because everyone else is looking at that person,” says Batra.

Get Arguable with Jeff Jacoby in your inbox:
Our conservative columnist offers a weekly take on everything from politics to pet peeves.
Thank you for signing up! Sign up for more newsletters here

The program still has limitations. In one test image, it identified a graduate’s tie as a person’s face, though, to the algorithm’s credit, it determined the “tie-person” to be of low importance. The engineers hope to make the program more discerning by adding in the ability to detect social cues (A white dress means bride! The person holding the scissors at a ribbon-cutting ceremony is important!), and by making it more attuned to a photographer’s intent (photographers following the “rule of thirds,” for instance, place their subjects slightly off-center).

In addition to organizing images by importance, the engineers anticipate their invention could be useful for auto-cropping and for creating automatic descriptions of images. “You would like the algorithm to focus on the important activity,” says Batra. “You don’t want the algorithm to say there is a man in the background with gray hair.”

Of course, any ranking system can be exploited, which creates one more potential application for the program — as a hyper self-aware party game in which people vie over the course of an evening to see who can rank as the most important person in the most pictures.

Kevin Hartnett is a writer in South Carolina. He can be reached at kshartnett18@gmail.com.

Correction: An earlier version of this story incorrectly referred to Dhruv Batra’s title. He is a professor of electrical and computer engineering.

Loading comments...
Real journalists. Real journalism. Subscribe to The Boston Globe today.
You're reading  1 of 5 free articles.
Get UNLIMITED access for only 99¢ per week Subscribe Now >
You're reading1 of 5 free articles.Keep scrolling to see more articles recomended for you Subscribe now
We hope you've enjoyed your 5 free articles.
Continue reading by subscribing to Globe.com for just 99¢.
 Already a member? Log in Home
Subscriber Log In

We hope you've enjoyed your 5 free articles'

Stay informed with unlimited access to Boston’s trusted news source.

  • High-quality journalism from the region’s largest newsroom
  • Convenient access across all of your devices
  • Today’s Headlines daily newsletter
  • Subscriber-only access to exclusive offers, events, contests, eBooks, and more
  • Less than 25¢ a week
Marketing image of BostonGlobe.com
Marketing image of BostonGlobe.com