How did the polls get it so wrong? From the US election to Brexit, polling is setting up voters for a fall

After one of the most extraordinary elections in history Donald Trump has become the most powerful man in the world, storming to victory to beat Hillary Clinton to become US president. People both in the US, and around the world, are processing the result which has come as a shock to many.

Throughout the campaigns and until the very last days all appeared to suggest a victory for the Democratic politician by a comfortable margin. Just last week, Ms Clinton experienced a small polling wobble as FBI action over alleged emails put her one per cent behind Trump. However, an announcement from the FBI that she was no longer being investigated saw her regain a comfortable foothold in the polls with a solid six per cent lead.

So how did the polls get it so wrong? Just as with Brexit, the British 2015 general election and the Scottish referendum, the polls have set voters and pundits up for a fall. Polling is major business around the world, with political campaigns pumping huge funding into them in a bid to gain a strategic advantage over rivals and prestigious universities and academics investing considerable time and intellectual effort in developing and backing particular models.

In the coming days, months and years, the polls which underestimated Trump will be poured over as statisticians scrutinise how and why they failed so spectacularly to spot the remarkable result. It will play a major part in the story of this election and the how polling is perceived and performed. The following four particular issues may be subject to the most scrutiny.

Firstly, results are still unfolding, so it may be the case that Ms Clinton wins the popular vote overall yet lost the election by electoral college votes. If that does happen, it may be that pundits overly focused on simple nationwide data, while failing to take into account the electoral college system. Simple binary statistics evaluating Clinton v Trump as a whole may be overly relied on by the media due to a need to communicate simple and accessible stories to readers who would otherwise be turned off by complex data sets, yet they can also obscure the full picture.

Secondly, a “shy voter” phenomenon is clearly emerging as we have seen among shy-Tories in the 2015 British general election and shy-Brexiters in the recent EU referendum vote. Politics is increasingly becoming more bitter and divisive as greater extremes emerge as a result of recent economic crash and unparalleled social change. Similarly, social media has made people more aware not only of how other people outside of their daily experiences or social groups think but also of the backlash which expressing opinions contrary to mainstream or socially acceptable discourse can accrue. Recent polls have successively underestimated support for less socially acceptable candidates or options, while overestimating support for mainstream candidates. A considerable number of voters may tell pollsters they are undecided or voting for what they think is the socially acceptable option, while planning to vote for a more controversial candidate who is their personal preference once they enter the polling booth.

A third failure which pundits will want to consider over the coming days and months will be methodology or how pollsters find voters and talk to them about their voting intentions. Traditionally, polls previously took place by pollsters knocking door to door and asking voters face-to-face. With the advent of telephones, this took on a difference dimension decades later. With the advent of the internet data collection options have changed again in recent years by opening up options of online polls and surveys, with varying accuracy. It is widely acknowledged that online data tends to oversample younger, more progressive voters and pollsters must make calculations to compensate this. However, as polls consistently overestimate support for liberal candidates this may not be adequately assessed and the balance between face-to-face interviews, telephone polls and online media may require finer tuning.

Finally, one explanation may be that many major political issues are simply becoming too close to call. Despite their best efforts, most pollsters admit that their research carries a margin of error by at least a couple of percentage points. This is to factor in human error, statistical flukes, poor methodology and other issues which can skew data. The final result of Donald Trump’s expected victory is yet to be announced, however he is expected to win by a couple of per cent rather than a landslide. Similarly with Brexit, while polls anticipated a slight Remain victory, they ultimately lost by a mere 48 per cent to 52 per cent, within a margin of error.

One thing however is certain: In order for polling to regain respect among voters and relevance among social science research it must urgently address these issues and provide some answers for future elections.