Labour in crisis: Is it really all over for Brown?

The early evening sun had all but disappeared from the Downing Street rose garden by the time Gordon Brown arrived for an end-of-term drinks party last Tuesday. Only the tops of the trees were still bathed in sunlight, but, for a rare moment in his difficult leadership, the Prime Minister's face appeared illuminated by the prospect of a turnaround in fortunes.

He had just returned from Israel, where his standing as an international statesman was boosted by a well received speech on the Middle East.

Early indications from the run-up to the Glasgow East by-election in two days' time were that Labour might just hold on. And minutes earlier, the Prime Minister had escaped being superglued to the hand of a Plane Stupid protester at another No 10 reception – prompting jokes by some present that, like his predecessor, he had finally become "Teflon" Gordon.

As he chatted to groups of journalists, Mr Brown recalled last year when his holiday was cut to just four hours because of a foot-and-mouth disease outbreak, which had followed summer floods and the terrorist attack on Glasgow airport. This time, he predicted, it would be a "quiet summer".

Now any chances of a revival seem to be extinguished, and he appears to be trudging towards the twilight of his leadership. As the Brown family settles today into its rented country mansion in Suffolk for a three-week break, he must fear the silence of a "quiet summer" – with nothing to distract him from behind-the-scenes plotting by cabinet ministers and disaffected backbenchers for a September coup.

It was not meant to be like this. Labour's high command had not expected the catastrophic defeat that befell them in Glasgow early on Friday morning, but they had made some preparations. The by-election, forced upon them when David Marshall unexpectedly resigned last month could have been delayed until the autumn. But Mr Brown chose to hold it early, to get any bad news out of the way so he could regroup over the summer – and, crucially, to make sure Labour backbenchers were starting their recess. MPs are a dangerous breed when they fear for their futures, but never are they so lethal as when they have time on their hands.

Tony Blair thought so. The former PM staggered towards the summer of 2006 a battered leader, after Labour had failed to regain the Blaenau Gwent seat at a by-election. The recess did not, however, bring any reprieve; by September, he had suffered a rebellion by MPs and ministers and been forced to announce that he would be gone within a year.

The Prime Minister might not face a rival with the smouldering presence of a Gordon Brown, around whom most of the 2006 rebels openly rallied. But the degree of concern about his ability to hold the party together is greater than anything Mr Blair faced during his darkest times.

"Summer is preparation time, September is when the fireworks begin," said one Labour source yesterday. "He's toast," a Blairite aide said, with some glee, recalling the mauling Mr Brown's predecessor took during his final months in office. "You can't help feeling that Gordon is going to get a taste of his own medicine."

Although MPs have officially been on holiday since Tuesday, the defeat – and the 22 per cent swing to the Scottish National Party – has dominated their communications since. Graham Stringer, the discarded minister routinely dismissed as "bitter" by Brownites, was predictably the first to go over the top, with the declaration that "we need a new start and that can only come from a debate around the leadership". But more have since become emboldened enough to join the charge, the latest being another former minister, George Howarth, who said yesterday that the required rethink "includes the question of the leadership of the party".

More significantly, however, Mr Brown's most senior colleagues already have their game plans in place. James Purnell, a star of the Cabinet, has agreed not to stand against David Miliband in the event of a leadership contest. Their fellow youngish gun, Andy Burnham, has followed suit. Mr Miliband, the Foreign Secretary, will not mount a challenge, but friends say if the opportunity arises there is no possibility of him ducking a second chance to go for the top job.

The pact, with echoes of the 1994 Granita deal between Tony Blair and Mr Brown, is part of the Blairites' revenge strategy, a plan to seize back control of the Labour Party and defeat a candidate such as Ed Balls. "I am not convinced that any of these guys would really want the job right now," one junior minister said. "But they would all jump if they thought someone else had stolen a march on them – especially if that person was Ed Balls."

Jack Straw, the éminence grise of the Cabinet, has been notable by his absence from the cameras during the Prime Minister's time of need, not least because he left on holiday yesterday. He has, however, made it known that he is working behind the scenes, urging more febrile colleagues to "calm down". Nevertheless, the Justice Secretary has emerged as the favoured choice of the left of the party to run as a "stop Miliband" candidate.

The most querulous harbour the hope that Mr Straw, along with chief whip Geoff Hoon, will agree to approach the Prime Minister to suggest he stand down for the good of the party. "Generous though that invitation is," a close ally of Mr Hoon's said, "I don't think Geoff will be taking it up." Mr Hoon does, at least, recognise that "a lot of Labour MPs will be concerned" about their party's predicament. "Nobody likes losing," he added. "The Labour Party recognises that we have got some tough challenges to face, but we are facing those challenges in the interests of the people of this country."

There is evidence that other senior ministers regard the Cabinet under Mr Brown as a sinking ship and are preparing to jump. Des Browne, the Defence and Scottish Secretary, is lining up a post-cabinet career in "conflict resolution", The Independent on Sunday has learnt.

His leader, meanwhile, is plodding on. An inspired speech on poverty on Thursday showed signs of flair and passion, but it was forgotten when the result came in from Glasgow. His speech to the National Policy Forum in Warwick went down particularly badly. One union negotiator said: "He made a 'steady as she goes' speech. But we have hit four icebergs and are sinking fast."

The truth is that Mr Brown and his senior colleagues foresee trouble ahead. "September will be worse than August," a senior government source said, "because most people will be away for the next few weeks. September was what did for Blair, because people started to come back and things were just as bad as they had been when they went away."

Mr Brown will, therefore, carry out a much needed reshuffle of his ministerial team in September, and he might purge those colleagues he had felt obliged to retain after the demise of the Blairite regime. The latest relaunch, another series of headline-grabbing policy initiatives rather than a dour restatement of his moral compass, will come during the Labour conference.

It will need to be good. Already, rivals are talking about stalking-horses and even "kamikaze" candidates, willing to take over in a desperate attempt to limit the size of an electoral defeat.

Brown allies are urging the leadership to call the contenders' bluff and circulate nomination papers for leader. Peter Kenyon of Save the Labour Party said: "This would reassert Brown's moral compass, draw a line under central command and control introduced by the Blairites, give Labour Party units an opportunity to back the leader or trigger a leadership election and demonstrate his political courage."

Or sign his own political death warrant. Mr Brown is facing the grim truth that he could not even win a John Major-style "put up or shut up" challenge. What might keep him in his job is the possibility that no one else wants it enough. "I think he'll cling on because there isn't enough appetite for change and because no one seriously wants to take the crap that he's getting at the moment," a loyal Brownite said. "I don't think that will last for ever."