From the general election to the rise of Jeremy Corbyn – it’s been a bad year for political predictions

He’ll come fourth. That’s what I said when Jeremy Corbyn scraped on to the ballot paper with one nomination and a minute or two to spare. He may yet, but I’m not quite as confident now as I was then. What is the point of political predictions, then? Especially mine. But the polls also got the general election wrong and no one saw the Scottish nationalist surge coming.

Predicting things is hard. But the urge to know what will happen next, which has driven so much of human culture, literature and innovation, is too strong to be deflected by the quietism of “whatever will be, will be”.

So we should respond to the failure of predictions not by dismissing forecasting, but by trying to work out why they were wrong and how they could be made better. Fortunately, help is at hand, in the form of a book published next month called Superforecasting, by Philip Tetlock and Dan Gardner. Tetlock is a giant of what he calls “the art and science of prediction”. He is the author of the famous study that rigorously recorded and checked predictions and concluded that the average expert was roughly as accurate as a dart-throwing chimpanzee.

In the new book he explains how he tired of the chimpanzee analogy, because it was so often simplified to “all expert forecasts are useless”. That is almost the opposite of his findings, which are that some forecasters are better than others. This book explores what it is about “superforecasters”, the ones who are consistently better, that makes them good.

So I studied the book for clues to why I had been so wrong about Corbyn – without realising, until I got to the “10 Commandments for Aspiring Superforecasters” at the end, that I was obeying number eight: “Look for the errors behind your mistakes.” That may not be a profound insight, but it is still worth doing in a systematic way.

One of my problems, I realise, is that I am not good at “belief updating”. This, says Tetlock, “can be boring, occasionally uncomfortable, but it pays off in the long term”. I now realise I was stuck with two assumptions that turned out to be false. One arose from the 1983 election, which was one of the formative events of my political views. That was when Labour offered a programme remarkably similar to that offered by Corbyn today (except that he has equivocated on pulling out of Europe, which Neil Kinnock did only when he became leader after the 1983 election). I assumed that the reaction to the defeat of Ed Miliband’s semi-skimmed Marxism would be similar, but that, because the Tory victory was narrower (a majority of 12 rather than 144), it wouldn’t take Labour 14 years to get back to winning elections.

The other belief that needed updating was that Labour Party members were pragmatists who were focused on the need to win. I knew that 54 per cent of members voted for David Miliband just five years ago, and thought that, even if the members had changed under his brother’s leadership, most of them would be influenced by the arguments for electability.

It is easy to see now that I should have noticed other things, such as the interaction of social media and anti-politics sentiment that made the SNP such a force. I knew that Mehdi Hasan and Owen Jones were popular, and that voices on the centrist wing of the Labour Party were not. I knew that the field of candidates was weak, and that the story of the election was so uninspiring that any novelty could have dominated it.

But then Tetlock allows me to avoid being too hard on myself. He says you should do postmortems on your successes too: and I did say since 2010 that I didn’t think Ed Miliband would win this year’s election, and hesitated only towards the end when the opinion polls sowed doubt. That failure of the polls has prompted some to suggest that YouGov’s survey of Labour members putting Corbyn on 57 per cent cannot be relied on. It is true that YouGov’s polling of party members wasn’t very accurate five years ago, being six points more favourable to Ed Miliband than the result. But even if YouGov has got it as wrong as it did then, Corbyn would still win on the first round.

Even if members and supporters rally around Yvette Cooper, who is now the party’s only hope, and shy away from Corbyn when the ballot papers arrive and most votes are cast this week, it is hard to see how she could make up enough ground.

And then what? I obey Tetlock’s first commandment and engage in “triage” – that is, to decide whether we should “waste our time trying to predict the unpredictable”. I do not know how Labour MPs and Corbyn himself will behave if he wins. But I still think we can be confident that Corbyn is highly unlikely to win a general election.

But I wouldn’t want to take all the unpredictability out of what might happen in politics over the next five years. That would spoil the fun.

Twitter.com/@JohnRentoul