Pity the pollsters who keep getting things wrong.
They didn’t expect Ayanna Pressley to best Michael Capuano in Tuesday night’s Democratic primary — polls showed Capuano with a comfortable, double-digit lead — yet she came out on top. By a lot.
It was the same story last week in Florida, when Tallahassee Mayor Andrew Gillum became the Democratic candidate for governor despite ranking fourth or fifth in pre-election polls. Or take the most iconic example of this election season, when the self-styled socialist Alexandria Ocasio-Cortez from New York leapt from obscurity to political stardom in June by upsetting one of the top-ranking Democrats in the House.
These progressive outsiders made the pollsters look foolish. And that’s no mere coincidence; it tells us something important about the limits of polling in 2018 and the wild uncertainty of the November midterms — despite the post-mortem that followed Trump’s unforeseen triumph in 2016.
For all of its mathematical rigor, the dark art of polling sometimes depends on what you might call educated guesswork. Particularly when it comes to questions of turnout.
Hidden behind those definitive-looking, topline poll numbers is a set of hard-to-test assumptions about who is actually going to show up to vote and who — among all the registered voters in a district — actually counts as a “likely voter,” the foundation of most polls.
This is a tricky thing to figure out, especially in primary elections and House races, when interest tends to be lower and local concerns more prominent.
Pollsters do generally ask people whether they plan to vote, but some lie and others don’t really know. And while a methodical polling group can try to pin things down by only counting folks with a real history of voting, that way they’re guaranteed to be wrong whenever anger or enthusiasm brings new people into politics. (This could explain Pressley’s surprise win, though it’s too soon to say definitively.)
This isn’t just a problem in Democratic primaries. Republican insiders have also lost races that pollsters thought were uncompetitive. In 2014, House majority leader Eric Cantor was dethroned, just weeks after releasing an internal poll showing him ahead by 34 points. Earlier this summer, Congressman Mark Sanford unexpectedly lost his seat to a more Trump-friendly opponent.
Yet, this time around it’s the Democrats who seem to be confounding polls as President Trump’s unpopularity scrambles political allegiances, activates new voters, and generally leaves pollsters short of the certainty they need to make good predictions of likely turnout, much less likely vote tallies.
One alternative would be for polling shops to publish multiple scenarios: one where they take a narrow view of who is likely to vote, another where they expect a rush of first-time and nontraditional voters.
David Paleologos, who directs Suffolk University’s polling center, suggested this could frustrate the whole purpose of polling: “People look to us for a clear picture of what’s going on, and it doesn’t serve them well if we provide multiple answers. We are expected to make a judgment based on our experience and the probability of what might happen.”
It doesn’t help that pollsters can’t get people to talk to them anymore. Once upon a time, survey groups could expect 80 to 90 percent of people to answer their home phone and share their views. These days, with social trust at a low ebb and mobile phones having replaced land lines, response rates are often below 10 percent, which makes it much harder to put together a good, representative sample. Internet polling is getting more sophisticated, but there, too, it can be hard to ensure a representative sample of respondents.
As a consequence, pollsters have to depend more on fancy math and rough-hewn assumptions.
Plus, there’s the oft-overlooked but perhaps most fundamental challenge with polls: They’re just snapshots. If you run a poll shortly before the election, only to find that the results don’t match the outcome, the fault may have nothing to do with methodology. People change their minds, right up to the last minute. Such late-deciders were a big reason Trump outperformed the polls in 2016.
A full month has passed since the most recent WBUR/MassINC poll of the Pressley-Capuano race, which is ample time for mind-changing. That’s particularly true when 15 percent of likely voters were still undecided at that point. Steve Koczela, who oversaw that poll as president of the MassINC Polling Group, thinks this was a key factor: “My sense is that the race really did shift, as Pressley became better known in the final weeks.”
Still, the most vexing problem of the moment is the search for likely voters. It’s a safe bet they won’t be the same people who voted in 2016 or 2014, not with Democratic enthusiasm running much hotter. But that’s too loose an understanding for pollsters, who need a much more concrete answer if they are to forecast the results.
Perhaps the real takeaway from surprise results like Pressley’s Tuesday victory is that some polls are likely to be very wrong right up until midterm eve because pollsters just don’t have enough hard data to model turnout in the Trump era.
As Koczela emphasizes: “We are always looking at our methods and trying to figure out whether this year is different from past years. But often you don’t know what you are doing wrong until it’s done.”
Consider that a warning for anyone hoping for pre-Election-Day certainty. Heading toward November, the country seems dotted with leftist outsiders like Pressley, Gillum, and Ocasio-Cortez, who draw their core support from unlikely voters and have cast their races as a quasi-referendum on a historically unpopular president.
Evan Horowitz digs through data to find information that illuminates the policy issues facing Massachusetts and the U.S. He can be reached at firstname.lastname@example.org. Follow him on Twitter @GlobeHorowitz.