John Rentoul: The Prime Minister's nightmare scenario

A by-election in Scotland combines all the elements of political ghastliness that Gordon Brown can imagine, short of Ed Balls actually coming into his office and demanding to know when he is going to be off.

The paradox of the Glasgow East by-election is that the stakes are higher than most people realise, yet the immediate impact of a Labour defeat is likely to be small. Despite Labour's fielding a relatively strong candidate and the Scottish National Party a weak one, Newton's Third Law of By-elections will probably hold: "For every chance to give the Government a kicking, voters will use whatever implement is to hand." The implement in this case is the SNP – and Gordon Brown will be duly kicked, a week on Thursday.

"You really need to hold on to Glasgow, don't you?" The Prime Minister appeared not to hear this question in his New Statesman interview last week. We can see why. Scotland is his home fixture, and the SNP are having the time of their lives.

A YouGov opinion poll on Friday put the SNP 4 per centage points ahead of Labour across the whole of Scotland. That wouldn't give Glasgow East, Labour's third safest Scottish seat, to the SNP in a general election, but by-elections are different. The SNP needs a 22-point swing to take the seat, and that sort of thing is normal in by-elections. Jim Sillars won Glasgow Govan from Labour for the SNP on a 33-point swing two decades ago. Labour were more popular than they are now; the SNP less.

Yet, in the short term, the consequences of an SNP victory next week are limited. The House of Commons will have begun its summer recess two days before. Labour's National Policy Forum, meeting in Coventry the following weekend, is a gathering of loyalists, activists and trade union officials rather than MPs and therefore not a good place to plot against Brown. Above all, defeat in Glasgow East would not be immediately dangerous to Brown because it has been discounted by MPs. They already know the situation is dire.

The story of New Labour's decline can be told in by-elections. It is a sad tale, with three exceptions. There were Labour holds in Sedgefield and Ealing Southall in the blissful dawn of Brown's premiership last July. And there was the diversion of David Davis's fight in an empty room last week, as he stood successfully against himself.

Brent East in 2003 and Leicester South in 2004 were Tony Blair's first losses, to the Liberal Democrats after the Iraq War. Four months before Brown took over, Labour lost another seat to the Liberal Democrats, in Scotland – worse, in Dunfermline and West Fife, next-door to Brown's own constituency. Then there was Crewe in May, the first Conservative gain since 1982, and safe Tory Henley last month, where Labour came fifth behind the British National Party.

What is most striking about Glasgow East is the desolation of the place. Britain as a whole may not be a broken society, as David Cameron claimed in a notable speech in the constituency on Monday, but this part is. Half the people on out-of-work benefits; broken families; drugs; low life expectancy. The place is a terrible reproach to Labour for the relative failure of its policies on social exclusion.

However, Labour MPs already know that their party is losing touch with voters, a process that began under Blair but that Brown has proved singularly ill-fitted to reverse. Losing another by-election adds little further information to the assessment of their chances at the next general election; nor does it provide the excuse to change leaders. For that, they need a policy that needs to be changed, and David Miliband does not have it yet.

Yet the Scottish dimension takes Labour's torment to a new level. An SNP strengthened by a by-election triumph has profound implications for the shape of politics to come. A Labour former cabinet minister told me recently: "My nightmare is that in 2010 the Tories win big, then Salmond has a referendum on independence, and we've lost the possibility of progressive politics for a generation."

For decades, the London con- sensus has been that the SNP is a regional protest party and that the Scottish people would never vote for independence. When the SNP formed a minority administration in the Scottish Parliament in May last year, it failed to disturb this assumption. I remember Blair describing the 2007 local elections, and by implication defeat by the SNP in Scotland, as "a perfectly good springboard to go on and win the next general election".

What he could have said was: "After me, the deluge." Because the situation looked very different in Scotland. The London consensus persistently overlooks the extent to which Scotland is a small, closed society, a bit like North Korea only less democratic. It has its own media and its own political culture.

It was not until Wendy Alexander, the leader of the Labour opposition in the Scottish Parliament, said on 4 May "bring it on" that most of us south of the border realised that there was a problem. She was challenging Salmond to hold the referendum on independence straight away. In Scotland, it made perfect sense, and was praised by the press there. In England, it made no sense at all. What was Gordon Brown's satrap in Edinburgh doing endorsing the main policy of the SNP? One Whitehall adviser suggested that Brown should say that Alexander's comment had been "mistranslated". Instead, he just claimed that she had not said it.

Alexander has since been forced out of office for a procedural irregularity that wouldn't raise an eyebrow in a residents' association. This may look like an illustration of Henry Kissinger's dictum about student politics – the infighting is so vicious because the stakes are so small. But the stakes are not small. Her departure is a symptom of the collapse of the Labour Party in Scotland.

That allows the SNP to fill the vacuum, and it is not inconceivable that Salmond could win a referendum in two years' time. His strategy is transparent, which was why Alexander tried to subvert it: to wait until a "Tory toff" prime minister is elected in England and hold a referendum on the back of three years of successful SNP "government". And it is going his way. Scottish opinion approves of Salmond's administration. Talking to ministers in London, it is apparent that even they regard the SNP as more competent and effective than the Labour shambles that went before.

So even though Friday's YouGov poll found that 57 per cent would vote No in a referendum on independence (excluding Don't knows), this is a margin that can be overturned. Of course, there is a barbed-wire tangle of complications between a Yes vote and eventual separation. Just to start with, there is public spending, oil and the armed forces, including nuclear bases. But one thing is clear: those of us who in 1997 dismissed the idea that devolution was a "slippery slope" towards separation got it wrong.

The former cabinet minister suffering nightmares is perhaps too partisan to see Scottish independence as ushering in a "generation" of Tory dominance in the United Kingdom of England, Wales and Northern Ireland. In fact, taking Scotland out of the House of Commons cuts Labour by "only" a net 27 seats – not enough to take away any of Blair's majorities, but enough to make it significantly harder for Labour to win again.

An SNP win in Glasgow would be another slip down the slope to separation. It may not prompt cabinet ministers to force Brown out in two weeks, but it ought to focus their minds on what is at stake in the next two years.