Just how wrong were the US election polls?

Yet another election ends with pundits telling you to never trust the pollsters. Are they right?

It must be tough working in the polling industry. When you get elections right (as polls did in the 2018 US midterms, or the 2019 UK general election), everyone moves on. When you get them wrong (as was the case in the 2016 presidential election, or the 2017 general election), you will own that failure for years. 

As initial results for this year’s presidential election came in, the outlook for polling appeared dire. Never mind a Biden landslide, Trump appeared to be winning re-election! However, as we all should know by now, those initial returns were heavily distorted. By the time most votes were counted, Trump had lost by a fairly decent margin. 

With near-complete results now in from battleground states, we can finally tell just how wrong the polling was (note: as I wrote here, the polls are always wrong to an extent. If they weren’t, we wouldn’t need to hold elections). 

Let’s get to it. The table below shows the final FiveThirtyEight polling average in each of the thirteen battleground states, alongside the actual result* and the size of error. 

*Some of these states are still counting, or are expecting recounts, but the final, certified, results are unlikely to differ by a meaningful degree. 

Source: FiveThirtyEight

When many people judge polling performance, they look at whether they predicted the correct winner. In this case, they did, not only nationally but in eleven of thirteen battleground states. 

Of course, this is not the right way to go about it. What we really need to look at is the size of polling error. To that effect, the polling in Wisconsin (which correctly predicted Biden’s win) was notably worse than in North Carolina (where it didn’t). 

One thing should immediately be clear: the polls underestimated Donald Trump in every key state. However, they did so to varying degrees. In Iowa, Ohio and Wisconsin, the size of error was very large indeed. In Florida and Michigan, polls were bad but not exactly terrible. In Nevada, New Hampshire, North Carolina, Pennsylvania and Texas, they were easily within the margin of error. In Arizona, Georgia and Minnesota, they were basically spot on. On average, across all thirteen states, polls were off by 4.3%. That’s far from an industry disaster.  

An important question remains. Why did the polling underestimate Trump across the board? Since 2016, one theory advocated for the existence of a ‘shy Trump’ effect, whereby supporters of the president were afraid to share their views due to a ‘social desirability bias’. Could this help explain the polling miss? 

Unlikely. In 2016, Trump tended to underperform his polling in blue states, where social desirability bias would presumably be at its most strongest. There was also no meaningful difference between live and online polls, which we would expect to see if a candidate’s supporters don’t want to be honest with a human interviewer. 

When it comes to 2020, there are other reasons to be skeptical. First, Trump underperformed Republican House and Senate candidates almost everywhere. Therefore, if the ‘shy Trump’ effect is real, there should be an even larger ‘shy generic congressional Republican candidate’ effect (which is unlikely). Second, Trump was doing notably better in the polls throughout the spring. This means that there would have to be a group of voters who were open in their support for Trump in March but suddenly became shy just a few months later (again, unlikely). 

There also happens to be at least one firm that made a conscious effort to control for the ‘shy Trump’ effect, that being Trafalgar Group. For months, Robert Cahaly, Trafalgar’s chief pollster, toured TV studious and proclaimed how traditional methods can’t be trusted. So how did they do? The table below shows the final Trafalgar Group polling in the ten battleground states which the firm surveyed during the closing weeks. 

Source: Trafalgar Group

In five of ten states, Trafalgar performed worse than the FiveThirtyEight average (this could become six by the time Pennsylvania concludes its count). While the margins are ultimately what matters, the firm’s methodological bias towards President Trump led them to miss Biden victories in Arizona, Georgia, Michigan, Nevada and Pennsylvania. To be clear, this is not a terrible performance for a single pollster (their final poll of Wisconsin was almost exactly right!), but also hardly a vindication of their methodology. 

There is another theory for why polling underestimated Trump support. According to David Shor, a Democratic strategist, it could be a result of differences in partisan response to the coronavirus. To that effect, Democrats were more likely to take the pandemic seriously while Republicans were less inclined to let it affect their everyday behavior. As a result, Biden voters, who were more likely to be at home, were simply more available to answer polls. 

Take the FiveThirtyEight average in Michigan, Pennsylvania and Wisconsin as of the 1st of March, before the coronavirus tore through the United States. In all three states, the figures are within two points of the final result. 

Now, this alone does not prove Shor’s theory. It could be that Trump was actually ahead back then, that the polls were always wrong, and that the pandemic lost him the election. This was before Biden became the presumptive Democratic nominee and before Republican efforts to tie him to the more radical elements within his party kicked in. There were also far fewer polls back then, making this hypothesis very hard to test outside of a handful of states. 

While there is far too much uncertainty at play here, this theory would be very encouraging if true. It would suggest that this year’s polling error was largely an anomaly, caused by a once in a century event. 

As in 2016, people far more knowledgeable in the subject than myself (such as the folks over at AAPOR) will conduct far more comprehensive evaluation. Once they do, I will certainly let you know. For now, I will leave you with one silver lining. Georgia, the battleground state that will, come January, decide control of the US Senate, is also the place where polls happened to be the least wrong.