The West Coast mainline, wasted taxes, and a secretive shambles at the heart of the Civil Service

A story that erupts from nowhere tells us more about how we are governed than the meaningless waffle from the party conference circuit. In the darkness of the night, the Government announced its decision to scrap the award of the West Coast Main Line franchise to FirstGroup. For the time being, Virgin will continue to run the trains, but no one knows what precisely will happen in the longer term. Companies have spent millions on making a bid. Taxpayers will pay them back. Perhaps new bids will be made if there is a fresh set of franchising rules with companies spending more cash in an attempt to secure the contract. Millions of pounds are changing hands, none of it being spent directly on train services.

The saga raises big questions about the accountability and competence of civil servants, the way we have chosen to run a national service and the culture of Cabinet reshuffles in which Transport Secretaries come and go more regularly than some trains. In each case, the questions and the answers have relevance beyond a single multi-million-pound train line.

Senior civil servants have considerable power and yet remain largely anonymous compared with ministers. Functioning in the dark, secure in their posts, remaining in departments longer than even the most enduring ministers, they show few signs of running the “ Rolls-Royce machine” of mythology. Successive governments are elected and express frustration at the inefficiencies and lack of grip in some Whitehall departments yet feel unable to do very much about it. When Prime Ministers are first elected, they have the authority to reform the Civil Service, but are inexperienced and reliant on the goodwill of officials so are wary of provocative reform. Later, Prime Ministers become unpopular and lack the will or confidence to act. There is never political space for substantial reform.

So the shambles over the West Coast Main Line, while shocking, is not especially surprising. For months, officials at the Department for Transport worked in untroubled isolation on the bids. There was little media scrutiny of what they were doing. Few knew who they were. In that protected darkness, they seem to have conducted a vital process with a casual lack of forensic urgency. Would they have done so if they knew some of them would be interviewed at ten past eight on Today or had to explain what they were up to in the House of Commons as if their lives depended upon it?

But officials cannot be held exclusively responsible. They were not operating in a vacuum. The current criteria for winning a bid, although impossibly complex in theory, are easily explained: the Government wants the “best bargain”. As the bids are considered, the main question posed is not: What is best for passengers and for wider transport priorities? It did not take a genius to appreciate that the sums from FirstGroup were not robust, requiring massive fare rises and a miraculous rise in passengers. Virgin’s Richard Branson made this point and can claim total vindication now. But there are doubts within the industry whether Virgin’s more modest targets could be met either. The franchising bids, where the rewards are great for the winner, inevitably tempt companies to make grandiose claims about their long-term plans, when the long term is safely years away. Who can blame them for claiming that they will transport many more passengers at reasonable fares, with lower subsidies, when such an offer can lead to the securing of a multi-billion-pound contract? They can worry about the long term when it comes.

All of them make their moves – the companies, the officials, the amply rewarded senior figures in the many other railway agencies – on the confident assumption that the Transport Secretary will be out of his or her depth. Under the last Labour government, Transport Secretaries changed about once a year. Only one of them, Andrew Adonis, had a passion for transport. Most had shown no previous interest or expertise. The same applies to the Coalition. To his credit, David Cameron has not changed ministers in most departments with quite the frequency of Tony Blair, but he has in Transport.

During the 10 minutes that Justine Greening occupied the post, she ratified the decision on the West Coast Main Line. I have just listened again to her Today programme interview on the day she made the announcement in August. During it, she insisted that the decision-making process had been exhaustive and extensive. She was wholly confident that the decision was the best for the taxpayer and passengers. What was she doing? She is an intelligent politician so I assume she accepted the advice of officials who had been working on this for much longer than she had, and that was more or less it. Perhaps she also felt pressure from the Treasury to deliver the offer that promised most cash even though it is now clear that the projections were a fantasy.

The lessons are wide. The review of franchising must include the most fundamental question of all: is the process unavoidably flawed? Among some right-wing ideologues, there is a shallow assumption that “involving the private sector” must always lead to savings. It can do in some cases, but in relation to the provision of a national service on which we depend, the “ involvement” can become impossibly complicated and expensive, with huge sums being spent on process rather than delivery. Franchising might work if the sequence becomes more transparent, but this is very difficult to achieve when private companies are bidding against each other and there is a need for confidentiality. If the lack of forensic rigour cannot be addressed, services should be renationalised when contracts expire, on the basis that this is the most efficient solution.

Senior civil servants must be more accountable, answering questions in the media, which, in modern Britain, is one of the ways public figures are held to account. They should be guests at ten past eight on Today. In particular, anonymous Permanent Secretaries should become better known, explaining regularly to us all what they and their colleagues are up to.

Prime Ministers should appoint Transport Secretaries who have an interest in transport. A Transport Secretary with such an interest should be allowed to serve for a full term.

Above all, taxpayers’ money, and the cash raised from laughably exorbitant fares, should be spent on improving train services and not on the bids, lawyers, accountants, regulators and agencies that are required when a national service is broken up to become a series of private monopolies.