Why do only failed politicians publish good books on their trade?

It is a rule almost without exception that the only good political memoirs or diaries are written by bad politicians. I do not mean malicious politicians, though malice can add to the quality of the writing; I mean politicians who were not very good at doing what they are supposed to do, perhaps because they were in the wrong profession.

One choice they have to make is whether they want to devote their careers to getting as high up as the ministerial ladder as possible, which is what most do; or sacrifice all that for greater political freedom of life on the back benches. Chris Mullin, whose diaries A View from the Foothills are published this week by Profile (£20), moved uneasily from one career choice to the other. He was doing very well as a Labour backbench rebel, treated with wary respect by Tony Blair, when unexpectedly called to the colours as a very junior and very frustrated minister for something or other.

His favourite anecdote from this ghastly period is the occasion when an invitation arrived in his office, passed on from the office of a more senior minister, with a civil servant's note still attached. It said: "This is very low priority. I suggest we pass it to Chris Mullin." He wrote "NO" all over it, and "within the hour" a civil servant was in his office to stress the importance of the invitation.

After two years, he gave up this life and returned to the backbenches, though it meant a £27,000 drop in his salary. He seemed to have burned his bridges when he voted against going to war in Iraq. Extraordinarily, Blair forgave him that rebellion, and brought him back to spend two years as a junior Foreign Office minister. He enjoyed his second bout in office, and being fired, at the age of 57, to make way for someone younger was a shock that he did not anticipate.

Indeed, on the day he was sacked he was so sure that he had survived that he had just rung his opposite number in the US State Department to discuss Liberia. Then "the Man" – the name Mullin always used in reference to Blair – came on the phone to exhibit his special mix of pseudo-friendly charm and cold ruthlessness. "The Man sounded remarkably cheerful. No hint of what was to come. We exchanged chit-chat ... and then came the fatal words: 'I'm sorry, Chris, but I am going to have to let you go'."

Here I must interject with a personal declaration. I have known Chris Mullin for longer than he has been an MP. There are political journalists who regard him as a hypocrite for allowing the serial rights of his work to be sold to the Mail, after the things he has said over the years about right-wing newspapers, but I have always liked him. There are some references in this diary which flatter me.

So, as I write that Mullin's diary is the best first-hand account of the Blair years so far, you will understand that I am biased. You will also know that the competition is not stiff. There has been a flood of books from departing ministers, including The Point of Departure from Robin Cook; The Blunkett Tapes; An Honourable Deception? by Clare Short; Momentum by Mo Mowlam and numerous others – with not a rivetting read among them.

Alastair Campbell's Diaries benefit from the many years he spent on the Daily Mirror, which trained him to write fast and well. Its drawback is that Campbell is too much of an insider. He does not look at Tony Blair and the inner circle in a way with which the rest of us can empathise.

New Labour's contributions to the art of diary writing make you almost nostalgic for the chaos and backstabbing of the Harold Wilson years, which produced fabulous chronicles by Dick Crossman, Barbara Castle and Tony Benn, Cabinet ministers all. Benn, that inveterate diarist, carried on scribbling for decades, but obviously any observation he makes from 1979 onwards is as an outsider, who moves further and further out as the years go by. He is perhaps more interesting writing from the outside than from within.

But let us not speak ill of the Labour Party. If you really want to read bad political memoirs, there are still remaindered copies of the blatherings of ex-ministers from the Conservative governments of 1979-97 gathering dust in second-hand bookshops, some so dire that you wonder how anyone in the publishing profession was ever persuaded to waste paper and ink on them.

Norman Fowler's memoir Ministers Decide is often held up as the worst of the bunch, which I always thought unfair. I think the critics picked on him because they knew he could express himself well if he wanted to. He has since written A Political Suicide – a very fine account of the disintegration of John Major's government. No, to say that Ministers Decide is the worst book ever written about the Thatcher government is to traduce the reputation of some of the truly monumental bores from that era.

I have before me Speaking My Mind by Rhodes Boyson and, honestly, if ever there was a mind best left unspoken, it was surely his. Or there is My Style of Government by Nicholas Ridley, compared with which Norman Fowler's insipid memoir takes on the pace of a thriller.

Amid the dross were a couple of good books by very high-ranking politicians from those years. Nigel Lawson's The View from No 11 is not for the faint-hearted, being very long and stuffed with statistics and economic argument, but anyone who gets through it will emerge far better informed. And Margaret Thatcher's The Downing Street Years is meticulously researched and well-written – a tribute, I suspect, to the high-powered team of researchers who assisted her.

Those are books for people with an absorbing interest in politics. If you are only half interested, but looking for an entertaining read, only two books from the Conservative era were in that class. Both illustrate the point that to write a first-class political diary you need to be a second-class politician.

One is Alan Clark's bestselling Diaries, the other Breaking the Code by Gyles Brandreth. It was a struggle taking Brandreth seriously as a politician, because he is better known as a wit, storyteller and collector of jumpers and teddy bears. But his detailed diary of the death agony of John Major's government is a delight.

At least there is little you can object to in Brandreth as a human being, whereas Alan Clark was a wealthy right-wing snob, a lecher and a borderline racist, who not only worshipped Thatcher the politician, but lusted after her too.

In her book, which runs to more than 600 pages, she mentions him just once, comparing his role in her downfall with the interlude in a Shakespeare tragedy when the clown comes on stage to bring light relief. That is the joy of the Diaries: Clark was a clown, who did not know he was a clown, who hung around the edges of power making a fool of himself, recording in anguish the failure of his career.

Chris Mullin has aimed his diaries at the same half-committed reading public. He describes in painful detail the frustrating life of a junior minister, with an outsider's eye. The most quoted passages have been the comical descriptions of John Prescott, his first departmental boss, who "has no concept of how to get the best out of people", but there is much more to be mined out of this volume.

I liked his take on that old story about Jo Moore, the New Labour political adviser who thought 11 September 2001 was a day to bury bad news. Another politician writing about that infamous email would have felt it necessary to offer his own opinion, probably to add to the thousands who had already passed judgement.

Mullin is only interested in teasing out what Tony Blair thought, when confronted by a group of Labour MPs who wanted Moore to be sacked. "The Man was having none of it. 'Too harsh. The end of her career – where would she find another job? She's got two kids, she's the main breadwinner.' I warmed to him. It reveals an attractive side of his character. Tony the Merciful."

That was bad judgement by Blair. If Mullin had been more of a politician, he would have seen that it was, but the passage is an amiable change from the usual bile directed at a woman who has suffered for her sin.

Unfortunately, the weakness of Mullin's book is that he is not a clown, but a self-aware character making the most out of situation which he chose to get himself into. For all the mishaps he suffered in government, Mullin never did anything to compare with being drunk at the Despatch Box, as Clark once was.

His diaries begin on the day he was first offered a government job, in July 1999, and end on the day in 2005, when he was sacked by telephone. Unfortunately those years threw up only one real drama, the invasion of Iraq, and during that episode he was honourably on the outside. So there is no story here to compare with the last days of Thatcher or the last years of Major.

The book is a series of tales, nicely written, from a fairly dull interlude in British politics. There will be a far better story to be told about what is happening inside the Gordon Brown administration. Sadly, there may not be a Clark, or a Brandreth, or a Mullin, in there to tell it.

Behind the political scene: the pioneers

Henry 'Chips' Channon

The original and still, for many, the best insider journal. A rich kid from Chicago, Sir Henry Channon became a UK citizen, married a Guinness heiress and sat as Tory MP for Southend from 1935 to 1958. Discreetly gay, a lavish West End host, fabulously indiscreet, he kept a famous diary that – drastically edited – appeared in 1967. In his own will, Henry's son Paul Channon – also a Tory politician – stipulated that the rest of the diaries should stay under wraps until 2018.

Richard Crossman

Life-long Labour Party apparatchik and maverick intellectual, Richard Crossman picked a pioneering fight with the Whitehall guardians of secrecy when he issued three volumes of his 'Diaries of a Cabinet Minister' in the 1970s. Covering periods as housing and health minister in the Wilson government of 1964-1970, they offered revelations and indiscretions that might seem tame enough now, but felt like political dynamite in that era.

Barbara Castle

A journalist when elected as an MP in 1945, Castle gained office in Wilson's government in 1964 and, until 1976, served as minister for development, transport, employment and health. She published her diaries after Thatcher's victory in 1979. Despite the big issues that occupied her in office – from railway cuts to union battles – she focuses most on the nitty-gritty of ministerial routine.