Andrew Grice: Big Society might work in rural areas but it's unlikely to play on estates

I went into the conference about the Big Society as a sceptic about David Cameron's flagship theme.

But I listened to the speeches was gradually being won over. Then Mr Cameron went and spoiled it all by saying something stupid like: "I've seen it action in my constituency where people have got together to clear up some common land."

Aha. His remarks, at an event just before last year's election, inadvertently confirmed my worst fears. The Big Society might work very nicely in leafy Oxfordshire, but what about the most deprived parts of the country?

Yesterday Consulting Inplace warned that Mr Cameron's big idea could result in a more divided society, widening the gap between well-heeled rural areas with high levels of volunteering and deprived cities in the North and Midlands. "This could mean the most vulnerable in society benefit the least," its report said.

The Big Society bombed when Mr Cameron put it at the heart of his General Election campaign; even loyal Tory activists admitted it didn't play on the doorstep. Cameroons argued that it would be an easier concept to explain once they were in power. Now one minister admits: "It was easier to talk about in opposition as an umbrella theme. It is much harder to translate into hard policies."

Ministers tie themselves in knots as they try to explain the apparent contradiction in handing out money to voluntary groups to become less reliant on the state. Symbolically, Lord Wei, the man chosen to head Mr Cameron's pet project, has scaled back the time he commits to the unpaid job because he needs to earn some money.

More dangerously, it has become a figleaf for spending cuts. The voluntary sector is being squeezed as the big cuts imposed on local authorities trickle down. It is squealing and the Government is starting to get the blame. One Cabinet minister told me: "We have turned what should have been a positive into a negative. The Big Society is starting to damage us."

Some ministers were never really committed to it. You won't hear Nick Clegg talking about it. Some Lib Dems believe the private sector will be much better equipped to land contracts to run state-financed public services than voluntary groups, making it look like back door privatisation.

The Liberal Democrats were committed to localism long before devolving power became fashionable with all parties. Indeed, the whole thing is hardly a new idea. As church leaders tell government ministers with a smile: "We've been doing it for 2,000 years."

Part of the problem is that Mr Cameron has not defined it. Flak is now flying retrospectively at Andy Coulson, former Tory director of communications, who preferred bread-and-butter issues like law and order and was at loggerheads with Steve Hilton, the director of strategy and chief evangelist of the Big Society.

The idea has become all things to all men. At the moment it looks like the label is stuck on to every passing Government policy in the hope it will catch on. Plans to sell off England's public forests were foolishly and belatedly marketed as part of the Big Society on the grounds that charities or local groups might take over some woodlands. It gave the big idea a bad name.

Ed Miliband believes the Big Society is Labour's natural territory, but prefers to call it a Good Society in which the state and communities join forces but the state does not withdraw. He believes Mr Cameron does not understand the value of communities – and would not allow the closure of libraries, cuts in police numbers or the privatisation of forests if he did.

However, Blairites think Mr Miliband should resist the temptation to say the Big Society is all about cuts, which won't help Labour regain its economic credentials or do the party any good in 2015 if the economy is on the mend. They want Mr Miliband to outflank Mr Cameron on his home turf by saying he should go "further and faster".

Timing is all and Mr Cameron's big idea might have worked in the years of plenty when the Government could have oiled the wheels properly. It is not a good idea in an age of austerity.

But all is not yet lost, the Cameroons insist. Despite all the problems, the Prime Minister will not let his pet project die a death by a thousand spending cuts. He will come out fighting on Monday as he seeks to redefine it and there will be a flurry of Big Society initiatives next week. "He will make this work," one Cabinet ally said yesterday. "We are in choppy waters but he is the PM and the machine will have to deliver it. He is not going to drop it."