Andrew Grice: To overcome his Downton problem, David Cameron should start by claiming the nation as his own

When voters are asked what they associate David Cameron and his Government with, some reply: "Downton Abbey". One Tory Cabinet minister admits: "The programme comes up in the focus groups."

I'm sure it's not because of Mr Cameron's "upstairs not downstairs" background. Most people wouldn't mind a privileged prime minister if they felt he governed fairly. The reason people mention Downton Abbey is surely that the Government's actions have made it appear out of touch with ordinary people and as if it looks after its own. It started with the March Budget, and a cut in the 50p top rate of tax.

Mr Cameron was warned by modernisers, including Andrew Cooper, his director of strategy and polling expert, that it would be politically dangerous. The Prime Minister stopped George Osborne reducing the rate to 40p from next April, limiting the cut to 45p. But the damage was done. Even if Treasury figures eventually show higher tax revenues, the political effect of cutting the tax paid on earnings above £150,000 a year when millions of people feel their living standards stagnate, will remain toxic.

Other elements of the Cameron project to detoxify the Conservative brand have also stalled. Mr Cameron, having promised no more top-down re-organisations of the NHS (his top personal priority), launched a top-down re-organisation of the NHS. The Chancellor has diluted the Coalition's commitment to the green agenda, to the horror of the Liberal Democrats. "All we've got left is gay marriage," groaned one disillusioned Tory moderniser.

Cameron aides were happy to see last month's Cabinet reshuffle portrayed as a shift to the right in the hope that it would bolster the Prime Minister's position among his rebellious, right-leaning, coalition-hating backbenchers. Andrew Mitchell, appointed Government Chief Whip in the shake-up, quickly reinforced the Tories' posh party image by allegedly calling policemen guarding the Downing Street gates "plebs".

Many voters would probably be fairly relaxed about being governed by "born to rule" Tories if they governed competently, especially in an economic crisis. But this lot have not done so in recent months. The Budget unravelled spectacularly, prompting several U-turns. A new word, "omnishambles", was even coined to sum it all up. Despite Downing Street declarations about turning a corner and moving on, the fiasco over the franchise for the West Coast rail line shows that the shambles continues. You can hardly blame Ed Miliband for grabbing the One Nation banner and running off with it. The Tories have lurched right, giving Labour and the Liberal Democrats an opportunity to fill a gap left by the now-defunct Tory left led by Kenneth Clarke, a general without an army. Mr Miliband's impressive landmark speech, which electrified his party's Manchester conference, would never have worked if Mr Cameron had not given every sign of deserting the centre ground on which he had pitched his tent on becoming Tory leader almost seven years ago. Arguably, it is the Tories who have unintentionally re-ignited the class war, not Mr Miliband. Political attacks have to be credible to work.

When Labour ran a "Tory toffs" campaign, top hats and all, in the Crewe and Nantwich by-election in 2008, it backfired badly. Why? Because the public had clocked that Mr Cameron was trying to "detoxify" his party at the time.

True, One Nation is hardly an original idea. Mr Cameron and Nick Clegg have both claimed the mantle. A re-energised Mr Miliband will need to keep up the level of performance he gave this week until 2015 rather than disappear off the radar again. He is adamant that One Nation is not a five-minute wonder, that it is a route map for his party as well as a destination for the country. But he will need policies to tell us what it means. And he can't continue to face both ways on issues on which Labour is vulnerable, such as welfare and immigration.

New polling funded by Lord Ashcroft, the Conservatives' former deputy chairman and major donor, shows that Labour is ahead of the Tories on "standing for fairness"; being "on the side of people like me"; "representing the whole country, not just some types of people"; sharing "my values" and ensuring "equal opportunity for all".

However, the Tories are ahead, by a 2-1 margin, on being "willing to take tough decisions for the long term". Indeed, the Cameron-Osborne team is convinced that Labour's Achilles heel is that it would increase borrowing. Those focus groups may agree with Labour that the Government is cutting "too far, too fast," but people run a mile when told that Labour's answer to the debt crisis is to borrow even more. Indeed, the Prime Minister and Chancellor are attracted by the idea of re-running the Tories' 1992 "tax bombshell" election campaign in 2015. Funnily enough, Labour's best brains have also turned to the 1992 campaign recently. They say the party will not make the mistake it made then by setting out specific tax and spending plans – and handing ammunition to the Tories. It was fired by the then Chancellor Norman Lamont and his youthful bag carrier, David Cameron, then seen as pretty Thatcherite.

Who is the real David Cameron? Is he the moderniser who transformed the Conservatives' "nasty party" image, the man who had a photograph of the One Nation Tory PM Harold Macmillan rather than Margaret Thatcher in his office as Opposition Leader? Or has he now reverted to Thatcherite type under the twin pressures of right-wing Tory MPs and the need to cut the deficit?

Tim Bale, professor of politics at Queen Mary, University of London, and author of two books about the modern Tory party, concludes: "Cameron was and is ultimately no less a Thatcherite than the vast majority of his colleagues … Cameron, as Leader of the Opposition, never really engaged in any of the big arguments with his party that someone driving genuine change really needs to have."

Naturally, Cameron aides dispute that, insisting that his Conservatism is still rooted on the centre ground. Being attacked by his own party's right and by Labour on the left underlines that he is still in the man in the middle, they say. Although initially surprised by the strength of Mr Miliband's performance, Team Cameron judged that it was just that – and not matched by substance. It believes that Mr Miliband's speech signalled right but shifted left because of his mistaken view that the centre ground has moved leftwards.

Mr Cameron is at his best when his back is to the wall. That is now. He won't want to pay Mr Miliband the compliment of devoting his Tory conference speech on Wednesday to One Nation Conservatism. But he does need to answer the Labour leader's charge and reclaim the centre ground.

The Prime Minister believes he failed to win an overall majority in 2010 because he didn't finish the job of detoxifying his party. But that is no reason to give up on the project and lurch to right.

Most Tory MPs want more Thatcherism, not more modernisation – but I doubt the voters do. Next week Mr Cameron must resist the temptation to pander to his Thatcherite party and instead speak to the nation. If he does not, he will be a one-term prime minister.