John Rentoul: The day Boris became Mayor was the beginning of the end for Dave

That was when it all started to go wrong, thought David Cameron in the car on the way back from Buckingham Palace. It took only five minutes to return to Downing Street, but in that time he managed to blame at least four people for the collapse of his government and the fact that he had just become one of the shortest-serving prime ministers in history.

"It was all Boris's fault," he shouted at Sam.

"Yes, lover," she said soothingly. She'd never been keen on Boris Johnson anyway, because he wasn't very green. But now wasn't the time to say that she had told her husband so.

Cameron sank back in his seat. He remembered that night, four years ago, when all the bright young Tory things in his Leader of the Opposition's office had been so excited. Their party had actually won something. One young chap – he was an MP now, for somewhere in Kent – was breakdancing on the shadow cabinet table.

The ex-prime minister had always had his doubts about Boris. He remembered another night, much longer ago, when he had ducked out of the after-dinner entertainment of the Bullingdon Club at Oxford. He had sensed that trouble was brewing, and the following morning his fears were confirmed. He was regaled over breakfast with the story of Boris, trouserless, sprinting over Magdalen Bridge to avoid police interest in someone else's altercation with a plant pot outside a restaurant.

He hadn't wanted Boris to be the mayoral candidate in the first place. "That was Steve's fault," he said to himself. Steve Hilton, his director of conceptualisation, had the idea to have an open primary election to choose the Conservative candidate for London. It was postponed once because no one credible came forward and, after embarrassing conversations with Greg Dyke, Digby Jones and John Stevens, Boris was Cameron's only option. There had been a difficult meeting in Cameron's office which had been like trying to catch a greased pig. "Oath in blood of undying loyalty, shoulder to shoulder, that kind of thing," Boris had said, running his hand through his hair and staring wistfully across the river at County Hall.

Undying loyalty, muttered Cameron bitterly as he thought of all the trouble Boris had caused him even before the Olympic budget scandal had blown up in their faces. Mayor Johnson had gone ahead and cancelled the £25 congestion charge for gas guzzlers despite Cameron asking him not to. There was the time Dave had to force Boris to put out a statement saying that his denial of human-made climate change was "meant ironically". And there were all those well-sourced stories in The Daily Telegraph during Boris's early honeymoon as mayor about his using City Hall as a platform for his bid for the Conservative leadership.

But the real damage was done by the reintroduction of hunting for urban foxes in the Greater London area. After that, 74 per cent of people agreed that Tories were "toffs who think they are born to rule". And there were the Tofftastic viral videos on YouTube that got so much play in the 2010 general election campaign.

His first election, he thought. How unlucky he had been! He had enjoyed that last golden year of Gordon Brown's premiership, when Rupert Murdoch decided that the young Tory leader "can walk on water after all". He uneasily recalled his airy confidence in the happy year of 2008, saying that it was an "outside chance" that Labour would change its leader again, and that it would make matters worse if it did.

He had reckoned without the declining but still strong instinct of discipline among Labour MPs. He had not anticipated the steel of David Miliband, the Foreign Secretary, in wielding the knife. He was still kicking himself for buying the conventional wisdom that it would make the Labour Party look ridiculous if it changed its leader twice in one parliament. Of all people, he should have known how such assumptions about leadership contests are there to be overturned.

But he was proud of the way he had responded to the changed situation. He had kept his balance and prevented his party from talking itself into a downward spiral. Miliband had pulled the Government back from the depths and Chancellor Ruth Kelly's Budget had been a clever one that enabled Labour to go into the election with the narrowest of opinion-poll leads.

As Cameron looked back, he could not think of how he could have fought the 2010 campaign better. If only Boris hadn't brought the Tube network to a halt with the strike over the no-strike deal right in the middle of it. "Dave, Dave, it's all right. I'm calling their bluff," Boris had said on a crackly mobile from a safe house. "If they go ahead, they'll get the blame; if they fold, we win – it's the Battle of Navarino all over again."

Sam, as ever, had been right: "Like as if."

The car was gliding through the rephased traffic lights at Parliament Square by this time and Cameron saw a bendy bus in the queue being held up by the police outriders. He burst out: "And he never even managed to get rid of those stupid buses!" Samantha put her arm round him and said, "Stay cool, babe," as she smiled and waved at the crowds, mostly tourists.

Still, he had been a good prime minister in difficult circumstances for the past two years. Admittedly, he hadn't been as prepared for the complications of a hung parliament as he should have been. He should have paid more attention to Peter Kellner, president of YouGov, who predicted Boris's election as mayor.

Around that time, George Osborne had been waving about Kellner's article in Progress magazine on the five hung parliament scenarios and how different they were, saying: "We must war-game these."

They hadn't, of course. So when the Conservatives ended up with fewer seats than Labour and short of a majority even with the Liberal Democrats, there had been "a bit of scrabbling around the minor parties", as he had famously been overheard saying. All those secret meetings with Chris Huhne turned out not to be worth the paper the Green Compact was written on. It was Huhne's coup against Nick Clegg, of course, that had finally convinced Labour that a change of leader could lift opinion-poll ratings rather than incur public ridicule.

All the same, as the prime minister of a minority Conservative government, Cameron had surprised himself with the maturity of his judgement and the skill with which he managed the Commons. The government had some solid achievements. Frank Field had been a great success as welfare reform minister. The economy was just beginning to respond to Osborne's stimulus package, too. But it was too late. "It's all Gordon Brown's fault," Cameron said out loud, startling Samantha.

As her husband resumed his reverie, she was left to wonder at her forgetting that Brown had ever been prime minister.

"President McCain called to offer his sympathy," said the private secretary in the front.

"What about President Blair?" asked Cameron. "I've just fought an election saying I want to save his Eton academy from Labour's dinosaurs, and he's on the phone to Miliband congratulating him on getting back in." He was still smarting over Miliband's reinvention of himself during the campaign as a left-wing populist.

As the car turned into Downing Street, the awfulness of his "collapse of stout party" during 2012, ending in defeat in that second election, bore down on his greying head. If only he had gone to the country last year, as he had originally planned, he thought. But the outbreak of the new strain of bluetongue had put paid to that.

So he had been trapped. Kate Hoey, the minister for the Olympics, had warned him that there was something wrong with the numbers back in January, but he had been distracted by the fight with Boris over "ending Londoners' subsidy to Scotland". The warning bells should have sounded when he heard Boris boasting about "a humungous cost under-run", and saying the country was going to make a £4bn profit from the Games. But then it was too late, and before he knew it Boris was denouncing the idea that he put pressure on the auditors to delay publication as "a pinko liberal conspiracy of intergalactic proportions".

Cameron had hoped that Boris's defeat in May by Rushanara Ali, the Labour MP for Bethnal Green hailed as Britain's female Obama, would lance the boil of the Olympics scandal, but the media simply switched the blame to the government.

Then came the bolt from the blue. The pre-Budget report last month had been a triumph. Then Alex Salmond announced that the Scottish National Party would vote against the government unless it reversed its deal with Boris to take "London's money" from Scotland. "Wily Salmond wins again," crowed the Scottish press as Cameron was forced to go to the country.

Cameron got out of the car at No 10 and turned to wave at the cameras before getting on his bicycle for one last ride back to Notting Hill.

"Anything to say, sir?" shouted Paul Lambert, the BBC producer who had followed him like a stalker throughout the election campaign.

"Goodbye, Gobby," said Cameron cheerily. Under his breath he muttered: "Bloody Boris."