farah stockman

Big Data’s big deal

The power of pattern in collective human behavior

istock photo/globe staff illustration

A few years ago, New York City set up a “geek squad” tasked with analyzing vast amounts of data to improve the way the city runs.

It used information from utility companies to identify which buildings lost power after a storm; data from sprinkler systems to find those most vulnerable to fire; geospatial data from sewers to figure out which restaurants were clogging pipes with illegally dumped grease.

New York is at the forefront of the “Big Data Revolution,” which could usher in one of the greatest societal changes of our lifetime. Vast amounts of digital information are being put to creative, problem-solving use. Google can help predict flu outbreaks by mining data about where people are looking up symptoms. Real-time information from commuters’ smart phones can help urban planners design better transportation systems. The number of UPS packages can detect an upswing in the economy.


It’s important to remember all the good things that can be done with Big Data at a time when so many are focused on the bad. Since former National Security Agency contractor Edward Snowden exposed secret US government programs that collect information about phone calls and websites, metadata has become a dirty word.

Get Arguable in your inbox:
Jeff Jacoby on everything from politics to pet peeves to the passions of the day.
Thank you for signing up! Sign up for more newsletters here

The truth is, we still don’t know whether those programs are mining metadata for patterns of behavior. Fred Cate, director of the Center for Applied Cybersecurity Research at Indiana University, says that if they are not, “they should be.”

The FBI and CIA have always mapped social networks to figure out who is talking to known terrorists. Big Data makes that process faster and more detailed. It can also generate clues in a cold case.

“There is an old joke about the FBI investigating a lot of pizza delivery places,” he said. “People in hiding tend to have food delivered, and make a lot of calls for pizza.”

The idea of the government snooping in our data without permission has understandably aroused outrage and fear. Some worry about a return to the bad old days of the 1960s, when the FBI tapped the phones of civil rights activists.


But according to Victor Mayer-Schonberger, co-author of “Big Data: A Revolution That Will Transform How We Live, Work, and Think,” that’s not really data’s dark side. The real danger, he said, is not so much spying on a single individual, but rather the transformation of an individual into a piece of a vast pattern that can be used to predict events.

“What we need to fear from Big Data is not necessarily old-fashioned surveillance, but probabilistic predictions that punish us not for what we have done, but what we are predicted to do,” he said.

As good as it may sound to use data to prevent crimes by predicting who will commit them in advance, that kind of activity crosses the line and subvert the most fundamental principles of justice.

Even so, companies already do something similar: Visa blocks your credit card if you make a purchase too far afield from your usual routine. Geico determines your level of risk, and your car-insurance rates, using algorithms that analyze your accident history — along with those of millions of other people your age.

So the issue is not really how to protect privacy in the age of Big Data. Privacy, in the old fashioned sense, is already gone. The question now is: How can we be sure that the Big Data out there on us will be used only for good?


Neither Mayer-Schonberger nor Cate had a neatly packaged answer, but both agree that far too much energy is spent hand-wringing about who can collect data, for which purpose, and whether they have gotten consent. Those questions become almost meaningless. Collection is happening already, everywhere, at a rate we can’t control. So let’s focus instead on creating good rules for how Big Data can be used.

“If I use that data to save your life, you are not going to care how it was collected,” said Cate. “But if I use that data to track you down, then it is going to bother you.”

Farah Stockman can be reached at fstockman@globe.com. Follow her on Twitter @fstockman.