SAN FRANCISCO — David Soloff is recruiting an army of “hyperdata” collectors.
The company he cofounded, Premise, created a smartphone application now used by 700 people in 25 developing countries. Using guidance from Soloff and his co-workers, these people, mostly college students and housewives, photograph food and goods in markets.
By analyzing the photos of prices and the placement of such everyday items as piles of tomatoes and bottles of shampoo and matching that to other data, Premise is building a real-time inflation index to sell to companies and Wall Street traders, who are hungry for insightful data.
“Within five years, I’d like to have 3,000 or 4,000 people doing this,” said Soloff, who is also Premise’s chief executive. “It’s a useful global inflation monitor, a way of looking at food security, or a way a manufacturer can judge what kind of shelf space he is getting.”
Collecting data from all sorts of odd places and analyzing it much faster than was possible even a couple of years ago has become one of the hottest areas of the technology industry. The idea is simple: With all that processing power and a little creativity, researchers should be able to find novel patterns and relationships among different kinds of information.
For the last few years, insiders have been calling this sort of analysis Big Data. Now Big Data is evolving, becoming more “hyper,” and including all sorts of sources. Start-ups such as Premise and ClearStory Data, as well as larger companies such as General Electric, are getting into the act.
A picture of a pile of tomatoes in Asia may not lead anyone to a great conclusion other than how tasty those tomatoes may or may not look. But connect pictures of food piles around the world to weather forecasts and rainfall totals and you have meaningful information that stockbrokers or buyers for grocery chains could use.
And the faster that happens, the better, so people can make smart — and quick — decisions.
“Hyperdata comes to you on the spot, and you can analyze it and act on it on the spot,” said Bernt Wahl, a fellow at the Center for Entrepreneurship and Technology at the University of California Berkeley. “It will be in regular business soon, with everyone predicting and acting the way Amazon instantaneously changes its prices around.”
Standard statistics might project next summer’s ice cream sales. The aim of people working on newer Big Data systems is to collect seemingly unconnected information such as today’s heat and cloud cover and a hometown team’s victory over the weekend, then compare that with past weather and sports outcomes and figure out how much mint chip ice cream mothers would buy today.
At least, that is the hope, and there are early signs it could work. Premise claims to have spotted broad inflation in India months ahead of the government by looking at onion prices in a couple of markets.