Steve Richards: The roads to reform are paved with good intentions

There is a distinct echo as the Coalition's rushed revolution slows down. Until the late winter it was all systems go as senior ministers and advisers sought to transform the role of the state and its relationship with the private sector. Now we still await the White Paper on this theme, and specific policies aimed at realising the vision are a contradictory muddle.

The echo comes from the recent past. Tony Blair sought to follow the same route. "At our best when at our boldest," he declared as he turned to the private sector and the markets to revive public services. He retreated too.

Why is it that prime ministers embarking on this particular route move with such tentative and confused hesitancy after proclaiming their intent with crusading resolution? In both cases the excuses are similar. Always the advocates insist it is to do with the politics and not the substance of the reforms. Currently those getting the blame for David Cameron's retreat include the Liberal Democrats, the Treasury and the European Union. Under Blair, Gordon Brown was blamed as the "roadblock to reform", or for being "anti-reform".

These convenient scapegoats are almost universally accepted as being the reason why no further "progress" is made. And yet a closer examination shows that the actual causes for the familiar retreat are deeper. After all, in both cases the similar programmes of reform had the backing of prime ministers with great presentational gifts and the almost wholehearted support of the newspapers – a powerful alliance.

In the case of Blair, towards the end of his leadership he had the political freedom to go as far as he wanted. It is true that in the second term Brown sought to block some of his reforms. But towards the end of the second term and during the third Brown was so neurotic about being portrayed simplistically as the roadblock to reform that he gave Blair a free hand. Blair's biographer, Anthony Seldon, describes this period as "Blair Unbound". Still Blair was fairly bound, not by Brown but by the limits of what could be achieved through competition and market-based reforms.

The case of Alan Milburn is emblematic. Milburn was a dynamic Health Secretary with a vision that had the enthusiastic backing of Blair, most of the cabinet and, again, every newspaper. He resigned partly because of Brown's round-the-clock resistance to some of what he was trying to do. But if he was so sure he was following a route that would succeed, and so determined that his vision should be realised, could he not have stayed on in an attempt to prevail? The reformers who share this particular vision do not seem to last very long in power. Look at what will happen to Andrew Lansley. Now there is speculation that Steve Hilton, Cameron's policy guru, might leave No 10 in a state of disillusionment, although the reports reflect more frustration than a definite decision to leave.

It cannot be a coincidence that these particular reformers become frustrated when there is so much going for them in terms of initial prime ministerial enthusiasm, media support and a pervading sense that they are on the fashionable wing of politics, the crusaders for "change".

Why do they face obstacles? The answer is that there is less in these market-based reforms than meets the eye. They have merit in some circumstances, but are not the all-embracing, coherent, efficient solution that the evangelists and indeed orthodoxy suggest. That is partly because the term "reform" is itself confused. To take one example of many, this crowded branch of reformers often hails localism as the answer, and yet city academies, arguably the most substantial change to have been implemented, are an act of centralisation in which schools are accountable to the Education Secretary.

Competition can and should be used to shake up complacent public services, but the reformers apply their faith too indiscriminately, without any proper analysis of whether markets can work effectively in services that are a monopoly, such as the railways, or in health, where demand is different and less predictable than weekly shopping. They assume always that the private sector is more efficient than the public alternative, although evidence suggests that the reality is much more mixed.

To make these obvious points is not to be "anti-reform". There is more than one set of reforms. As David Willetts has acknowledged, most of these fashionable "reforms" do not address the supply side, how we get better teachers, nurses, doctors and, in some cases, how we get more of them. Nor do they address how we get them in the right places. The senior Liberal Democrat Julian Astle has argued astutely that breaking up national pay bargaining to take more account of ability and cost of living would make a greater impact than any other change in getting the right nurses and teachers in the right places.

The issue of ownership is also underrated. Those looking at co-operatives and the so-called John Lewis model move on to fruitful terrain. Unavoidably, the centre that raises the money for services also still has a powerful role to play. In this context the reform of the civil service is urgently needed. Senior civil servants are often in departments longer than ministers, and in their durability are more powerful. They should be held more to account for the delivery of services, along with those lower down their multi-layered hierarchy.

Instead, all political capital is spent repeatedly on bringing in the private sector to deliver public services. Earlier this year Cameron wrote a restless article, arguing, "We need a complete change... The grip of state control will be released". In the same article he added, as if it was a mere tiny qualification, that "the state will still have a crucial role to play: ensuring fair funding, ensuring fair competition, and ensuring that everyone – regardless of wealth – gets fair access".

If applied properly, he was still defining a massive role for the state, even when his revolutionary zeal was at its peak. That is the boundary these particular reformers come across. Their reforms have a role, but a limited one when an elected government has many unavoidable obligations. They can blame politics, the Treasury, the EU and other parties. In the end they must face up to the mundane truth that there are limits to their reforms and they are always forced to do so.

s.richards@independent.co.uk