Andrew Grice: What's Labour got to lose by axeing Brown?

"It won't happen at the Labour conference. It won't happen when MPs return to Westminster. It could happen around the turn of the year, if he doesn't turn things round."

No prizes for guessing what the cabinet minister was talking about. The assassination of Gordon Brown is again the subject of intense speculation in his own party. And not only among the usual suspects.

This minister, speaking anonymously and from the heart, is regarded as a Brown loyalist, but reckons there's a 60-40 chance that the Prime Minister will be ousted before the general election, after his poor start to the new political season.

The Labour conference in two weeks will probably rally round Mr Brown, who plans a personal speech rather than his usual dogged defence of everything he has ever done. He will not go down without a fight, and is working his usual 17-hour day as he plots his attempted fightback.

There may be some noises off in Brighton. A coded debate about Labour's future has begun, which is dangerous because it virtually concedes that the election is already lost. "Some people are starting the inquest before the defeat," snapped a Brown ally, referring to this week's opening salvo by Jon Cruddas, darling of the Labour grassroots, who said Labour had offered "no compelling case for re-election".

Brown allies grumble that some cabinet ministers appear more interested in preserving their reputations in their own field rather than taking the fight to the Tories. "Some people are settling for a quiet life," said one.

Team Brown also tears its hair out over how to swim against a media tide flowing strongly in David Cameron's direction. If Mr Brown had made a speech about putting up the price of MPs' beer, he would have been ridiculed. Yet the Tory leader's speech about cutting £120m off the cost of politics was widely seen as a serious contribution to reducing this year's £175bn deficit. Supporters of regime change believe that a new leader – Alan Johnson, the Home Secretary, is the man most likely – might just halt the tide, allowing Labour a hearing for the choice between the two main parties.

Backbench knives are being sharpened for the Commons return on 12 October. New wheezes to destabilise Mr Brown are being dreamt up. But his MPs have tried twice before and failed.

Labour's emerging consensus is that Mr Brown's point of maximum danger will come in December or January – if he has not improved his performance and/or closed the opinion poll gap with the Tories. This would be Labour's last chance to change its leader before the election, which must be held by next June. Waiting until December could mean that the Micawber Tendency finally runs out of excuses, and would defuse Tory and media demands for an immediate general election. If a new Labour leader were not installed until January, he or she could swiftly name a date in April or May, taking the wind out of Mr Cameron's sails.

There is another reason why December may prove the cruellest month for Mr Brown. Labour's hard left and the trade unions are the dogs that have not barked. The assumption is that they stick with him for fear of something worse, and calculate that their best hope would be to exploit a backlash against New Labour after an election defeat.

I am told that their mood is now changing. Some left-wing MPs and union bosses are coming round to the view that they would have an overriding duty to avoid a massacre that could keep the party out of power for a generation."If nothing has changed by December, we would have nothing to lose," said one prominent left-winger.

So Mr Brown's fate may well be sealed by the public, via the opinion polls. If so, he has only a couple of months to make an impact.

As ever, the final push would probably have to be made by senior cabinet ministers – the missing ingredient in the two backbench plots. There is renewed talk of a "graceful exit" and "bloodless coup" in which Jack Straw, Alistair Darling, Harriet Harman and Lord Mandelson persuade Mr Brown to stand down for the sake of the party.

True, they have stood by him before, and may do so again, on the ground that no one else would do any better at the election. On the other hand, there are signs that they might just grab that final opportunity. There is even high-level chatter about Mr Brown taking an international job to cement his place in history as the man who saved the world in the financial crisis. Which job? "We can always invent one," one senior figure told me. I laughed. He wasn't joking.