Why did journalists get it so wrong on the election? Even when they spoke to real people, they didn't believe them

During the 2015 election campaign, all the talk was of the strong possibility of a hung parliament. The country, it was said, had had enough of austerity.

Two years later, there were predictions abounding of a vast Conservative majority. The Tories, wise heads argued, would be backed overwhelmingly as the only party capable of undertaking the difficult Brexit talks.

In between the two elections was the EU referendum, which most predicted (even if they hoped otherwise) would result in a vote to remain. And the less said about how Donald Trump was written off before his presidential triumph, the better.

All this raises some serious questions both for the media and for psephologists – and for the interplay between the two. That is particularly so now, given that such a lot of promises were made in 2015 about journalists no longer being slaves to opinion polls. Instead, history repeated itself.

For the pollsters, failings during the 2015 campaign led to an in-depth investigation commissioned by the British Polling Council and the Market Research Society. The report concluded that researchers had systematically over-represented Labour supporters and under-represented Conservative followers; and had not done enough to adjust the raw data they collected. Notably, the authors of the report recommended that pollsters should “review existing methods for determining turnout possibilities”.

A couple of years on and there is much speculation that it was on the question of turnout that polls erred once again, with more 18-24 year-olds voting on the day than pollsters had anticipated. Only YouGov, which predicted a hung parliament in its eve-of-election poll, seemed to get the measure of the electorate – and even here its predictions in Scotland were fairly wide of the mark.

What about the media, though? After all, journalists ought to hold all statistics in sceptical regard. Why the blind spot when it comes to political polls?

To an extent, of course, it’s not a question of blind spots but of seeing the numbers as drama. The predictions which come thick and fast through an election campaign might not be right but they are a story in themselves. When the positive impact of Labour’s manifesto seemed to be reflected by a narrowing of the Conservatives’ lead in the polls, the narrative which had a whopping Tory landslide at its heart began to be questioned. Even if few believed Theresa May would not win a majority, the hint of shifting public opinion injected excitement into the media’s coverage.

If too much stock was placed in the opinion polls, perhaps too little attention was given to the views of real voters. Or more to the point, journalists often listened to the views of individuals but then presented them through the prism of their own preconceptions. Thus, Labour voters at Jeremy Corbyn’s rallies might have been wildly enthusiastic but they didn’t represent the wider public; meanwhile, ex-Ukippers might not think much of May but they’d vote Tory to ensure that there was no back-sliding on Brexit. A few writers saw which way the wind was blowing – but too many ignored its strength if not its direction.

The other factor which seems to have stumped the majority of political journalists was the apparent confidence among the Conservative rank and file. For all the divisions over the social care U-turn and for all the grumbling about May’s decision to avoid both tricky media appearances and real voters, all the mood music from the Conservative party itself was positive, right up until election day.

Looked at in the cold light of day, there are countless assumptions which now look utterly foolhardy. First, commentators – especially those over the age of 35 – had too little faith in young voters actually turning up at the polling stations. For 20 years, those in the 18-24 age bracket appeared to have given up on mainstream politics; but the success of the Momentum movement to engage younger people should have pointed to the bucking of that trend.

Second, we appeared to forget just how hated the Conservative party is in some parts of the country. The presumption was that once a Labour voter had switched to Ukip, shifting to the Tories would be easy: that proved manifestly not to be the case. For some, voting blue is simply beyond the pale.

Perhaps most obviously, an assumption was made that if voters weren’t prepared to say no to austerity in 2015, there was no reason for them to do so now. But that indicates both a misreading of the outcome in 2015 – when many votes for Ukip were a cry against the broader status quo, not only against Britain’s EU membership; and an underestimation of how hard life is for a great many people in this country. The so-called “Just About Managing” were all the rage six months ago; they seemed to be forgotten about during the election campaign, except by Labour.

So here are some lessons many of us in the media need to learn. Don’t trust polls; don’t make assumptions; don’t believe party apparatchiks know something no one else does. Most importantly, get out and talk to real people – and then believe what they are saying.