The police have been exposed over plebgate. Now give Mitchell his job back

There are many eerily precise echoes with the 1970s in the tumultuous era we are living through. Once more there is a hung parliament, a frightening economic crisis and a referendum being held in Scotland. David Bowie is huge and Paul McCartney has released another album. There is however one very big difference.

Whereas then there were power cuts and three- day weeks, now the lights are switched on, at least metaphorically. Their beams are focused on previously sheltered institutions. Suddenly we know much more about what has been going on behind the scenes at the banks, parts of the NHS, within newspapers, the BBC, some schools and the police. We know too that when greed, misconduct or breathtaking incompetence is exposed, the instinct from within such institutions is often to protect those caught in the light.

The revelations have been disturbing and in some cases shocking. In relation to the police, where light has been cast on a range of areas from their role in the Hillsborough tragedy to the fall of Andrew Mitchell, the beams expose especially disturbing issues. More than any of the other flawed institutions, the police can only function when its integrity is beyond doubt.

Yet the integrity of the police is being questioned at the highest possible level. Until recently political leaders worked on the assumption that they could not be seen to challenge the police in public. When Margaret Thatcher directed her reforming zeal towards the public sector she avoided the police, giving officers big pay increases instead. The last Labour government sought reforms but rarely made much progress, fearing any perception that they were anti-police and therefore “soft” on crime. The Coalition has been much braver, seeking to make the police more efficient. In doing so it has provoked even greater resistance from the police. Andrew Mitchell was an early victim of the battle.

Now the focus is on the police rather than Mitchell. This represents an almighty leap. Although David Cameron and his Home Secretary, Theresa May, were commendably bold in their attempts to reform the police, they remained until this week publicly polite and supportive, working on the assumption that at all times leaders and the police must appear to be at one. Now both Cameron and May have called for an apology from the police officers who misreported a meeting they held with Mitchell. They seek also an explanation from West Mercia police who concluded that the officers had no case to answer in respect of their meeting with Mitchell.

Fortunately Mitchell recorded the meeting and has evidence to show that the police officers lied about their exchanges. The fact that Mitchell felt the need to record the event is in itself a reflection of an extraordinary breakdown in trust. Mitchell was still a cabinet minister fighting to save his political career when he met the relevant police officers. With total justification as it turned out, he sensed that the officers at the meeting would distort their conversation.

Given the taped evidence, Cameron and May had no choice but to speak out. Nonetheless the distancing with the police is without precedent. In the same way that Gordon Brown felt compelled to be associated with the glamorous success of senior bankers in the era when the banks functioned in the dark, and then felt equally compelled never to be seen near them when light was shone on what they were up to, Cameron breaks a taboo in relation to the police.

I argued last year that Mitchell should have kept his cabinet post. After the disgraceful “cash for honours” police investigation, and following revelations about the the close relations between senior police officers and journalists in the various hacking sagas, my instinct in relation to Plebgate was to assume that at the very least Mitchell’s version might be close to the truth. More than a year later Mitchell deserves much more than an apology from the police. He should be brought back to the cabinet.

That is the easy part of the story, although I do not underestimate the impact on Mitchell of his career being brought abruptly and unfairly to a halt. The much bigger question is harder to answer: Why is it that suddenly a whole range of institutions is being plunged into crises relating to integrity and competence? The answer is partly due to tiny factors, such as the Freedom of Information Act that enables outsiders to find out what is happening. But the more fundamental explanation, I suspect, goes back to the 1970s and the extreme reaction to the chronic failures of that decade.

At that time the state seemed to be everywhere, setting the price of bread, subsiding companies that made goods nobody wanted to buy, agreeing wage levels. In response, the long-serving Tory and Labour governments stepped back too far and did not dare to act in any way that had an echo with the dark decade. Banks were lightly regulated. The debate around publicly financed institutions was the degree to which they should be set free to run themselves operate without any interference from government.

The latest manifestation of this fashion was the introduction of elected police commissioners. On Wednesday, Ron Ball, the commissioner for Warwickshire, appeared on Newsnight and appeared comically out of his depth. Ball has no previous policing experience and was elected to his surprise on a turnout of all of 15.65 per cent. This is not a robust form of accountability, and predictably the issue of what happened when Mitchell met West Mercia police officers renders a figure such as Mr Ball an irrelevance.

The banks were reckless. Some hospitals regarded the patients as obstacles who were in the way of an easier life for staff. The BBC paid too may senior managers preposterously high salaries and pay-offs. A few newspapers hacked phones. In reacting against the stifling corporatism of the 1970s the momentum moved too far in the opposite direction. The freedom to innovate lapsed too often into a freedom to abuse new privileges and responsibilities.

Now all of us are empowered with the information as to what has really been going on in the dark. The intense level of scrutiny is a wholly positive development. The more challenging issue is how best to respond.

It will not be long before we hear that politicians should not interfere with the police, the same argument used by some newspaper executives in relation to press regulation. But ultimately it is for elected politicians to set regulatory frameworks that hold powerful institutions to account. No one else can or should attempt to do it. In most cases self- regulation does not work. The lines of accountability in public institutions must be clear. If they are blurred cock-ups, or much worse, will continue to happen.

The lights are on but political leaders across the spectrum are dazzled by the unaccustomed glare. When they are used to it they must ensure that greater scrutiny means higher standards from those who preferred to operate in the dark.