Data scientist has been called the sexiest job of the 21st century. As organizations wrestle with the ocean of data generated by mobile devices, social computing, the cloud, and other technologies, data scientists help turn this information, known as Big Data, into insights companies can use.
Andrew Schwartz is a data scientist and cofounder of Lattice Engines, a data analysis firm in Boston. He lives on a sailboat on Boston Harbor; rides a bright, green recumbent bike to the office, and does his best work between midnight and 4 a.m. “To really understand a problem,” he says, “it helps to be completely isolated with my computer.”
Is there an art to data science?
Data science is a mixture of business analysis, analytics modeling, and data management. It’s like a scientist painting a picture: you have to do some analysis and sketch out the main figures while also teasing out fine-grain details. A data scientist needs to be methodical but also make guesses.
Data scientists look at data and spot trends. What trends have you discovered?
In a previous company, I worked to help a casino optimize slot machines. We discovered that the size of the “take’’ is not as important as the frequency and distribution, meaning people are very responsive to hearing someone else win, even if it’s just a small amount. They are more likely to stay around and play if they hear other people hit the jackpot.
How did you become a data scientist?
I have a PhD in math from Harvard. Like everyone else, I taught. I was standing in front of a calculus class of 150 students when I realized that none of the students really wanted to be there. After classes ended, I went sailing for three months with a programming book. By the end of my trip, I knew how to program.
What’s the future of the data scientist?
We are in the early stages of data science, so for the next five to 10 years, we’re going to be seeing a lot of projects that are addressing the types of problems that you need Big Data to handle.
Give us an analogy of how big Big Data is.
It’s equivalent to the total of every e-mail sent or received by Gmail for the past four years or every financial transaction by every bank.
Are you using data analysis to buy a new boat?
Yes. I’m following classified ads and doing a data analysis for pricing, including how long a boat has been on the market, and the relationship between price, space, and retail value. My boat broker hates me.