Yes, I got it wrong where Corbyn was concerned – but that doesn't mean the opinion polls were off-target

Enough of the self-flagellation. I have done it myself. Now Ian Katz, editor of BBC Newsnight, has got out the knotted rope, asking: “Why did (almost) everyone call the election wrong (again)?”

What does he mean, “almost”? I am told that when Jeremy Corbyn assembled his inner circle in his Islington home to watch the exit poll on election night, he asked them all to write down what they thought the result would be. All of them, including Seumas Milne, his director of communications, predicted an increased Conservative majority – except Karie Murphy, his office director, who predicted a hung parliament.

I accept that Murphy, an ally of Len McCluskey, general secretary of the Unite union, is the most brilliant analyst of British politics, and I congratulate her on her good judgement. But I do not know how she knew.

The final opinion polls – the worst way of finding out how people will vote apart from any other method – diverged widely this time, from a one-point Tory lead to a 12-point lead, but it is only with hindsight that we can analyse why Survation and the YouGov model were closest to the 2.4-point margin of the result and ICM was furthest away.

You could, as I did, take the average of the polls, an eight-point lead, which suggested the Conservatives were heading for a majority of 50 seats. You could be pretty sure that this would be wrong, but in which direction? I said at the time that, for the YouGov model to be right, “something would have happened in the past two years that makes the national opinion polls more wrong in a pro-Tory direction than they have ever been”.

Well, we know now that several things happened, most importantly that more people under the age of 35 turned out to vote than expected. But no one could be sure of that before the election. Even YouGov didn’t believe its own model: the company continued to run conventional polls in parallel, with its final one suggesting a seven-point Tory lead.

Labour MPs didn’t know. They were canvassing the wrong people: people who had voted Labour in the past and who were therefore recorded on the party’s databases. That meant they came across a lot of people, especially working class voters, who disliked Corbyn. They missed a lot of the young, middle class people who were coming over to the party.

Even the people running the exit poll for the broadcasters didn’t know until late in the day. I am told that in the 2010 election, the first wave of exit poll figures at lunch time pointed to a Conservative majority, and successive waves gradually went towards a hung parliament – which was the result. In 2015, the first wave was about the same, but this time there was no movement during the day – so the published prediction was close to the result again. So this time, when the first wave came in pointing to a small Conservative majority, the number-crunchers didn’t know if the numbers would shift towards a hung parliament (as in 2010) or remain constant (as in 2015). As we now know, the Tory numbers came down during the day and at 10pm the exit poll pointed correctly to a hung parliament.

The only person Katz quotes who “called the election right” is Matt Turner, a blogger for a site called Evolve Politics. Again, I admire his skill and congratulate him on his judgement, but the only part of his method that the BBC reported was “sites like ours had our ear to the ground”.

So, yes, I take my hat off to Murphy and Turner for getting the election right, but for the rest of us mortals there was no way of knowing in which direction the opinion poll average might be wrong. We knew (from the opinion polls) that the abolition of tuition fees and Labour policies on the NHS, schools and housing were popular among the young, and that Tory social care plans were unpopular among the old. We could have guessed, therefore, that turnout would be higher among young people, and lower among the elderly, but to be honest I thought all that would do would be to offset the historic tendency of the polls to overstate Labour support.

Most of the people I knew who thought Labour would do better than the opinion poll average were ideological supporters of Corbynism whose reason for their prediction was that Jeremy was right. I have admitted that part of the reason for my scepticism about him was that I made the opposite mistake of assuming floating voters would, when it came to it, shy away from Corbyn’s past associations and his high-spend, high-tax policies, but I tried to keep to the evidence.

I don’t think it was possible to work out before the election that Survation, the YouGov model and Evolve Politics were right and most of the other pollsters and political analysts were wrong. And long may it be so. One of the joys of politics is that people are unpredictable.

Update 20 July: A reader has been in touch to say that Karie Murphy was consistent in her view of the election. In the week before polling day, she took the afternoon off to buy a dress to wear to accompany Corbyn to Buckingham Palace when he was invited to form a government.