Thorny research problems, solved by crowdsourcing
Faced with a tough data analysis challenge as he struggled to answer questions about how the immune system works, Dr. Ramy Arnaout of Beth Israel Deaconess Medical Center took an unusual step. He went beyond his circle of Harvard colleagues and beyond the expertise of fellow biologists; he turned to software programmers scattered around the world who had little expertise in the life sciences.
The result: A deeply biological problem — analyzing the makeup of genes that produce proteins involved in the immune system’s ability to identify microbes — could be rapidly and efficiently answered by a community of more than 400,000 computer programmers who try to solve competitive coding challenges posted on TopCoder, a platform used by big companies such as Google, Intel, and Facebook.
The case study offers evidence that academic research may benefit from adopting approaches more often employed by businesses, which already use prizes and crowdsourcing to solve basic problems that stymie their own research teams.
“If you look across the landscape and you see, where are the best computer science minds employed these days? Is it Facebook, Google, Microsoft, or some start-ups — or are they going to come to work in academia and work on a biology of cancer problem?” said Karim Lakhani, associate professor in technology and operations management at Harvard Business School and one of the leaders of the research.
Crowdsourcing approaches can allow researchers to access a broad pool of talent, Lakhani argues, at a time when the amount of data generated in biology and other fields is growing much faster than the workforce of computational specialists.
The experiment, described in a letter in the journal Nature Biotechnology on Thursday, suggests that academic researchers — even those in communities rich with expertise, such as Harvard’s — can sometimes benefit from thinking outside the box when it comes to solving hard problems.
The researchers carefully stripped Arnaout’s conundrum of all its biological elements, turning it into an abstract computational problem. The researchers then posted the challenge on TopCoder. They offered a total of $6,000 in prizes, and within two weeks, 102 people had submitted software code attempting to solve the problem.
Sixteen of the submissions were more accurate than code that Arnaout had written and worked on over the course of several years. Now, Arnaout is trying to integrate that code into his research program and understand whether it could be adapted to even more sophisticated types of data analysis, such as pinpointing mutations in the genes that instruct cells how to make the immune system proteins.
“If the guy next door can solve your problem for you . . . then by all means go next door, because it’s efficient,” said Dr. Eva Guinan, an associate professor of radiation oncology at Dana-Farber Cancer Institute and Harvard Medical School, who leads an effort to discover efficient ways to attack research problems.
But, she added, “going out to a broader community . . . that has an array of skills, perspectives, knowledge, experiences that you could never duplicate at home is something that we ought to be much more thoughtful about doing much more rapidly.
“That’s critical, rather than just getting hung up and beating our heads against the wall for a long period of time in a purist, I’m-going-to-solve-this-problem-myself” attitude.
The Harvard researchers have already set out a number of other challenges to the programming community, including asking them to come up with code that will help predict hot spots of medical need in Boston and help solve other big-data biomedical research problems in HIV and genomics.
Steve Bonadio, vice president of marketing for Waltham-based InnoCentive, a company that allows clients to post challenges to a community of online users, said that he was encouraged by the study’s findings, but that often the barrier to outsourcing problem-solving comes from within: the reluctance or fear of organizations to acknowledge that the best solution does not lie within their own walls.
“There’s very often a ‘not invented here’ syndrome — almost a cultural barrier to looking externally for problems,” Bonadio said.
“Those barriers have been eroding over the last few years, but I think it’s still a very real problem within a lot of organizations in academia.”
Despite widespread interest in using prizes to spur innovation — ranging from X Prizes to a report suggesting that prizes be utilized by the National Science Foundation, to expanded federal efforts to use innovation prizes — the authors were clear that today’s research operations could not just be replaced by prizes.
There is little danger of that now: A paper published last year by MIT economist Heidi Williams said that $1 billion to $2 billion of research and development is funded through prizes, compared with $55 billion that universities spent on research in 2009.
But Julia Lane, an economist at the American Institutes for Research, said it was important to remember that prizes are generally a bad way to motivate basic research. To offer a prize, people have to know what the problem is and set out the challenge.
“What basic research is about is discovery,” Lane said. “It’s not about predefined problems, it’s about discovering problems, and kind of the moment of, ‘That’s funny — I didn’t expect that.’ ”