fb-pixelThe problem with polling - The Boston Globe Skip to main content
OPINION

The problem with polling

There are many reasons US polling painted an inaccurate picture of a blue wave.

“I feel bad because I lied to you,” confessed Arlene, an Independent voter from Florida whom I had known for two years.

Arlene was part of a group of 500 citizens across the country who had agreed to participate in my project to understand the American voter. When I first interviewed Arlene by phone in 2016, she explained she disliked both Hillary Clinton and Donald Trump, but that she had decided on Clinton after she saw the “Access Hollywood” tape in which Trump spewed lewd comments about women.

In the subsequent two years, Arlene continued to criticize President Trump, although she admitted that, as a businessperson, she liked his tax program and his insistence on reducing regulation.

Advertisement



Then, in the fall of 2018, I got a phone call. “I am so sorry, but I just had to tell you, because I worry that I am messing up your data,” she said. “I actually voted for Donald Trump in 2016, but if my husband ever finds out, he will kill me — and you can’t tell anyone.”

Arlene is not alone. People don’t always tell the truth to pollsters. Some believe their vote is private and no one else’s business, and so they lie to rebel against the intrusion. Others, especially those who planned to vote for Trump this round, worried their answers would be shared and that they could be put into a database of deplorable Americans. Some didn’t really know whether they would vote or how they would vote, but they gave the questions their best shot. Or, there is Matt from Massachusetts, who said he told a pollster he would support ranked-choice voting, even though he didn’t really understand how it worked. “I just didn’t want to sound stupid,” he said.

There are many reasons US polling painted an inaccurate picture of a blue wave — with substantial margins — for the November election.

Advertisement



In market research terms, polls can be inaccurate if there are flawed “weights” in the sample, shoddily-constructed questionnaire design, or minuscule contact rates.

Weights are used to ensure that the sample has the right mix of people. A pollster might get responses from a sizable sample of 5,000 voters, but if they are all over the age of 60, that skews the results. Although pollsters know this, they are constantly challenged by what to account for in their samples. For instance, in 2016, pollsters determined that they had overweighted for education in their sample. It meant that if only 25 percent of voters in a state had a college degree but 50 percent of respondents had a college degree, the poll’s conclusions would not represent the electorate. Getting the weights right is both tricky and important.

Sampling methodology can be as simple as whether you sample any adults or instead base it on citizens who are likely to vote — or whether you survey people in person or online. As for questionnaire design, we know a bad question when we see one, such as the survey question sent to me by a friend last month, “Are you likely to vote and will you support Trump?” The choices were yes and no.

Contact rates are a significant factor. Three years ago, I traveled to the Quinnipiac University Polling Institute, a highly respected polling center. Interviewers worked out of 160 cubicles, making phone calls to voters. As I watched them, I asked myself, “Who answers the phone these days?,” and it made me wonder about the challenge of getting anyone to respond; telephone response rates are currently lower than 6 percent. A low response rate doesn’t necessarily mean an inaccurate poll in itself, but it does increase the risk of error.

Advertisement



Thus, there are many elements of how pollsters create their models and assumptions that can lead to inaccuracies. When those are coupled with respondents who have no stake in telling the truth, who worry about looking dumb, or who lie just because they resent constant telemarketing interruptions, it’s a recipe for polls that don’t serve us well. A recent article in the Globe reported a new study out of the University of Southern California: that QAnon supporters ruined the data by not participating in polls. The theory was that followers of this conspiracy theory were underweighted in the polls because they distrust institutions and are therefore less likely to respond to a pollster. However, if trust in institutions is the key factor in accurate polling, we should not limit the problem to QAnon.

Fixing this problem requires action on many fronts: larger sample sizes, new methods of getting data that build trust with respondents (such as not having an interviewer with a Boston accent working in Mississippi), and building new models that ensure the weights used for age, education, gender, and political party also include other factors. We need new ways of determining likely voters and better approaches to understanding undecided voters. In my research panel, I got the most trustworthy responses when I asked my voters to predict how others would vote — an indirect approach built on a method called prediction markets.

Advertisement



Here’s what we can be sure of: When voters are exhausted, or distrustful of institutions, or concerned about their privacy — or just worried that a friend or family member will discover how they voted — they are not going to put all of that aside for a stranger who calls after dinner. As consumers of polls, we need to remember that polls are not crystal balls. They paint a picture, but they are only as accurate as the willingness of participants to discuss how they will vote — and to tell the truth about it.

Diane Hessan is an entrepreneur, author, and chair of C Space. She has been in conversation with 500 voters across the political spectrum weekly since December 2016. Follow her on Twitter @DianeHessan. See her methodology at https://www.documentcloud.org/documents/5979231-Diane-Hessan-Methodology.html.