Hotels are ranked on TripAdvisor, books and consumer goods on Amazon, restaurants and services on Yelp. The wisdom of the online crowd sways countless decisions on where to dine, what to read, and whether to buy.
But is the crowd really that wise? Apparently not always, according to a study published Thursday that showed consumers’ ratings of comments on a news website were easily swayed by the positive ratings posted by others.
The research, coauthored by a researcher at the Massachusetts Institute of Technology, provides a cautionary tale for the social-networked age: Grade inflation is polluting the Internet.
The study showed that while people were easily, if subconsciously, nudged into overinflated enthusiasm and approval, they were not similarly susceptible to negative influences. That suggests that review sites may be skewed toward overly favorable ratings.
“The case where the wisdom-of-the-crowd effects work well is where each person brings their own observation and knowledge, however imperfect and idiosyncratic,” said Christopher Chabris, an associate psychology professor at Union College who was not involved in the research. “What this shows is when you don’t have that independence and everyone sees the history of other people’s opinions, you can get big biases in the outcome.”
The findings published in the journal Science surprised even coauthor Sinan Aral of MIT’s Sloan School of Management, who studies how social influences can affect decision-making and purchases.
The experiment was conducted on an unidentified online news website, where comments are voted up or down based on how good other readers think they are. Aral and collaborators randomly selected 101,281 reader comments, and as soon as they were posted, the researchers rated some comments with one up vote and others with one down vote. Some were left alone.
They found that this minor intervention could lead to a snowball effect.
Comments given an initial positive vote were 30 percent more likely to receive a very high rating versus those that were left alone, and had final ratings that were 25 percent higher than average.
Negative votes, on the other hand, were quickly corrected by the wisdom of the crowd.
“It’s a bit worrisome,” said Aral, an associate professor of information technology and marketing. “If you think even beyond ratings, if this herding behavior, or the tendency toward bubbles is true, it has implications for housing prices, stock market prices, and we need future research in other settings.”
The power of social influence to skew people’s choices has been clear in offline environments. In one famous experiment conducted in the 1950s, people were asked to choose which of three lines was the same length as another. If other people in the room publicly made an obviously wrong choice, many participants went along.
Online ratings seem empowering; after all, the aggregated feelings of dozens, hundreds, or thousands of people aren’t subject to the same limitations of an expert review, which reflects a single person’s experience and may be influenced by conflicts of interest and other unknown sources of bias. But the new study is part of a growing body of research showing online ratings can also be skewed and could be vulnerable to manipulation by the companies or people being evaluated.
In a paper posted online late last year, Michael Luca, an economist and assistant professor at Harvard Business School, analyzed Yelp restaurant reviews and how they evolve over time.
Luca found that simply averaging all the reviews a restaurant receives might not be the most accurate way to rate the experience of going to the restaurant.
That’s partly because prior reviews appear to influence the reviews that come after them — the same sort of peer influence that was found in the new study. In Luca’s study, reviewer biases could lead to ratings that were off by as much as a quarter of a star, in either direction.
He developed a way to weight the reviews so the overall rating better reflected the diners’ experiences.
“Every type of information source comes with a whole host of problems. For reviews, it’s not that this is a deal-breaker that the problem exists,” Luca said. “It’s just that this is the first time people are starting to think about the problem because the system is new.”
It’s difficult to say how strongly the new findings would apply to other areas. Even within the news site the researchers used, certain topics were more subject to social influence than others: comments on stories about politics, culture, and business were subject to overinflated positive ratings, whereas comments on stories categorized as general news, economics, information technology, and fun were not vulnerable to the enthusiasm of the herd.
Aral said he now is working to understand the decision-making mechanisms behind the biased ratings he observed — and has experienced himself.
Earlier this summer, Aral was eating at a restaurant in New York and went to leave a review on Yelp.
He had intended to award a three-star rating but saw that the last review had rated the restaurant much higher and praised the prices and salad dressing. He chose four stars instead.
The bizarre case of a Red Sox division title banner that went missing came to an end Wednesday when a Malden man who says he found it in Somerville brought it to Fenway Park.Continue reading »
As officials on Cape Cod grapple with how to prevent fatal attacks, researchers caution that nothing can fully protect swimmers from sharks.Continue reading »
A new poll shows a healthy majority of her constituents don’t think the Cambridge Democrat should jump into the 2020 race.Continue reading »
A woman was fatally injured when she was struck by a tow truck, spurring a man believed to be her son to allegedly stab the driver five times. The driver was in critical condition Wednesday.Continue reading »
Andrew Hawkins, an NFL receiver for six years and Gordon’s teammate for three, said the “ball all the time” environment here will be good for Gordon.Continue reading »
“Fahrenheit 11/9” is Michael Moore’s latest polemical documentary, and it’s pretty awful.Continue reading »
Republicans are seemingly marginalizing Supreme Court nominee Brett Kavanaugh’s accuser, a risky approach in an age of #MeToo and with midterm elections on the horizon.Continue reading »
Columbia Gas wants to dramatically speed up the replacement of 48 miles of pipes in the Merrimack Valley after last week’s disaster. But experts said that plan is unrealistic and potentially dangerous.Continue reading »
Back-to-back losses in the Bronx has given rise to concerns historic regular season will go for naught in postseason.Continue reading »