John Rentoul: Now it is when, not if, for Mr Brown

The result of the Glasgow East by-election was the worst possible for the Prime Minister and the best possible for the Labour Party, because it makes Gordon Brown's early departure more certain. The narrowness of the Scottish National Party's victory suggests that, under almost any other leader, Labour would have held the seat.

That means that there is something worth fighting for. If Labour had lost badly, its MPs could have gone on their holidays knowing that it would all be over in two years' time. If Labour had held on, they could have pretended to themselves that they might just be able to pull things back under Brown.

Now they cannot. By the pool, their BlackBerrys will buzz with worried colleagues. They can tell each other that a by-election is just a by-election: but they know that what really portends disaster is the state of the opinion polls. They should take two printouts with them to the sun lounger. The ComRes poll in The Independent yesterday asked people whether, "generally speaking", they think of themselves as Conservative, Labour or whatever. Although the Tories had a record 22-point lead when people were asked how they would vote in a general election, when it came to the question of with which party they identify, the Tories were a mere three points ahead. There has been, therefore, no fundamental turn away from Labour.

Last week's survey by Ipsos MORI describes the party's problem in more detail. It found that 21 per cent of the electorate are anti-Brown Labourites, who said: "I do not like Gordon Brown but I like the Labour Party." Half of them intend to vote Labour anyway, but the other half are the key target group: the 10 per cent or so of voters who are sympathetic to Labour, do not like its leader and currently intend to put their cross elsewhere. Their demographic profile ought to be studied carefully by any candidate who is serious about succeeding Brown. They tend to be young or, especially, middle-aged, with children and mortgages. They do not seem to be concentrated in any region or social class. They are Mr and Mrs Normal, the glue that held the New Labour coalition together. If Brown stays, they will become the glue that sets on the Cameron Tory coalition.

Getting rid of Brown may not help. Labour MPs know that there are downsides to changing leader again, but they also know that they could hardly be worse off. Glasgow East suggests that it must be worth trying.

They have to ask themselves the Peter Mandelson question, asked when Brown was being compared unfavourably with another younger Everyman in 1994: "Who will play best at the box office?" This time the choice is between Brown and David Miliband. The young pretender is growing in stature all the time and refreshingly normal. He must be the first Foreign Secretary to throw an American football (a gift from Stanford University on a recent visit with Condoleezza Rice) around his vast office.

My esteemed colleague Alan Watkins takes the view on page 57 that the Labour Party has been indulgent of unelectable leaders in the past and is "disinclined to act" again. I disagree. The ideological passions that threw up Michael Foot and Neil Kinnock have been spent, and self-interest is the safest guide to the future.

Ministers want to stay in government, and most backbenchers prefer to be on the right-hand side of the Speaker's Chair even if they know that they will never be ministers or never be ministers again. One MP who wants Anybody But Brown tells me that the Cabinet knows the mood among the other ranks. Before the recess started last week, the leadership was "the only subject of discussion" in the tea room and in the division lobbies. Cabinet ministers know that whichever of them brings Brown down will be greeted with more exclamations of relief than accusations of betrayal.

The Glasgow East result was never going to break the dam. It just adds to the weight of water behind it. They have to get rid of Brown, but they don't have to do it yet. Of course, every day that they leave it, Cameron grows stronger. "We want GB to stay as long as possible," a Tory staffer said to me happily last week, as they prepared excitedly for Barack Obama's visit.

But the timing is awkward, and self-interest cuts both ways. If Brown goes, his successor will have to promise an early election. Cameron was smart on the morning of the Glasgow East result to call for a general election now. Dave Normal, who wears shorts when it is hot and has his bike nicked, is trying to box in David Normal, would-be Prime Minister Miliband. A second change of leader without consulting the voters would be hard to defend, and a new leader would in any case not want to repeat Brown's mistake of seeming to bottle the chance to seek his own mandate.

For a few Labour MPs, delaying an election is more important than winning it. Fortunately for the party there are more MPs for whom the legacy of crude Blairism – that winning is all – overrides all else. Even if "winning is all" has now been scaled back to "not losing as badly as we might otherwise do".

That is why all the objections to change will evaporate over the summer. Some of my sceptical colleagues ask: What is the policy difference that would justify a new leader? It is a good question. It recalls the fall of Thatcher, the last time a governing party deposed a sitting prime minister, because her refusal to ditch the poll tax was a large part of her end.

But you do not need much of a policy difference. Not only are the ideological divisions within the Labour Party less intense than they used to be, but the differences between the two main parties are small. That makes issues of personality more important than perhaps they should be. As Obama reminded us yesterday, there are indefinable qualities of leadership that defy rational analysis. I think Miliband has got "it" and Brown has not, however unfair that may be.

"Gordon can do it the easy way or the difficult way," a well-placed Miliband supporter told me last week. "The only question is how much he wants to try to take the whole Government down with him."

As the Government will be going down with him anyway, it is worth a bit of personal unpleasantness now to try to avert it.

So in the autumn, or next spring, there is likely to be a Cabinet putsch that forces Brown to announce that he will stand down when a successor is elected. There is an interesting question, prompted by Tony Blair's lingering departure, about how long a gap he can leave between the announcement and the leadership election, so he may not go until he has served two years next June. But then Miliband will win against Ed Balls and, from the left, probably Jon Cruddas.

The news that James Purnell, the rising Secretary of State for Work and Pensions, will not stand against Miliband is significant, not so much for confirming what we thought we knew anyway but as evidence of the increasing sophistication of Team Miliband's media operation. I believe that Alan Johnson means it when he says he does not want the top job, and the idea of Jack Straw as the Stop Miliband candidate is not going to work: "caretaker" is not a platform for a leadership election.

A general election could still be some time away, therefore next autumn seems most probable. The Glasgow East by-election may have brought the date forward a little, but its main effect has been to make it more likely that Labour will at least try to avoid what would otherwise be certain defeat.