The Defence of the Realm, By Christopher Andrew

TThe fact that this is not so much "a" history of MI5 as "the authorized history" is underlined by its sombre black jacket, which gives it the appearance of a British government document. In a very real sense, it is. For although the book has been commercially published and Christopher Andrew is an academic historian, it was commissioned by MI5. Once he had been appointed to the post of official historian, the book was written on MI5 premises.

This is a centenary history, starting with the founding of the security service in 1909. In the following year, MI5 and the Secret Intelligence Service (MI6) were created as separate services. Andrew engagingly charts the evolution of MI5 through two world wars, the Cold War, and now the war against fundamentalist terrorists.

Inevitably, the agency's focus has shifted from counter-espionage and counter-subversion to counter-terrorism. What hasn't changed is its clandestine nature. The existence of MI5 was not acknowledged officially until 1979, when Mrs Thatcher unmasked Sir Anthony Blunt as the Fourth Man in a statement to the Commons.

MI5 has to tread a wire: it needs to be secret to protect parliamentary democracy, but is under pressure to be as transparent as possible. This has led to an uneasy compromise. Andrew was given unrestricted access to almost 400,000 files, but most are referenced only as "Security Service Archives" and closed to the rest of us.

This puts him in a difficult position: if he can't share his key sources, he must ask his readers to take his analysis on trust. We have only limited means of evaluating MI5's official view, as transmitted by Andrew, on various critical issues – such as the "Wilson plot" of the 1970s or the "Death on the Rock" episode in Gibraltar in 1988.

Publicity material for the book has filled newspapers with stories of double agents and courageous derring-do. But what also emerges from these pages is a fascinating picture of MI5 as a tightly-knit institution, thriving on its atmosphere of secrecy. "No one, not even our own families, should be told where we worked or for whom," was the firm instruction to a new employee in 1931. Morale sank after the end of the Cold War and the Good Friday agreement of 1998, which led to massive cutbacks. But after 9/11, there was a rapid expansion of staff and a renewed sense of purpose.

The Service had a choral society, which takes its name "The Oberon Singers" from Oberon's words in A Midsummer Night's Dream – "We are invisible, we will o'er hear their conference." It also had a cricket team and cricket imagery frequently appears in correspondence. "So the first XI of MI5 is to play the Mau Mau," commented the head of the Overseas Division when MI5 officers were sent to Kenya in 1952.

The Service did not advertise openly for recruits before 1997: until then, recruitment was based on personal recommendation. This was a narrow social group, many of whom had served in India or elsewhere in the Empire.

Male officers listed their recreations as cricket and hunting, while women were graduates of elite schools and universities. Women have always played an important role in MI5 and two recent Director Generals have been female – Stella Rimington, one of the first women agent-runners, and Eliza Manningham-Buller.

Right up to the mid-1970s, the post-war Service refused to recruit Jews on the grounds that a dual loyalty to both Britain and Israel might create a conflict of interest. This was "inexcusable", Andrew rightly observes. So too was the attitude to black people of Guy Liddell, Deputy Director General. "It was true," he told the Joint Intelligence Committee in 1949, "that niggers coming here often went to the C[ommunist] P[arty]." There was no doubt in his mind that "West African natives are wholly unfitted for self-rule."

Shockingly, the Service carried out secret surveillance of the colonial delegations which came to London to discuss terms for independence in the 1950s and 1960s. Andrew gives a disturbing account of the stealthy gathering of intelligence on the delegates attending conferences which negotiated the independence of Cyprus and Kenya. The Home Secretary, Rab Butler, cynically condoned these operations on the grounds that "obviously the product was of great importance and of great value to the government negotiators".

In most of the Empire, claims Andrew, MI5 contributed to a smooth transfer of power through the work of its liaison officers. But Guyana, where Churchill wanted to "break the Communist teeth", was a shameful exception. Here, MI5 supported British and American covert action to oust the democratically-elected Cheddi Jagan from power. Andrew claims that the Service was not "directly" involved and that the dominant intelligence agency in the years leading up to independence in 1966 was the CIA. But neither point exculpates MI5 or, more pertinently, the British government.

The Service has had some remarkable achievements, notably the Double-Cross System of the Second World War, which fed disinformation to the Germans. For the most part, however, it is difficult to measure MI5's success, since it can only be judged by things which do not happen – like the prevention of sabotage.

But it seems astonishing that it was not until 1951, as the result of the decrypt of a KGB telegram, that any of the Cambridge Five – all MI5 or MI6 employees, recruited at Cambridge in the 1930s to become spies for the Soviet Union – were identified. The decrypt took the Service completely by surprise and began the most drawn-out investigation in its history, taking over 30 years to complete.

The thousand pages of this book are brimming with some wonderful details. But many could be pruned – like the fact that the first Director's garden contained "400 rose trees and a grass tennis court". This would make room, perhaps, for a fuller account of the human factor in spying.

What was the intellectual and psychological motivation of the staff of MI5? – not only of the spies, agent-runners, and codebreakers, but those who steamed open envelopes and eavesdropped on telephone conversations. How did they feel about deceit?

The Defence of the Realm is a valuable and important contribution to our understanding of the 20th century. But an official history can only do so much, especially of an organisation that is inherently secret. In this hazy world of smoke, mirrors and lies – where actual conspiracies are barely distinguishable from conspiracy theories – we also need the scrutiny of genuinely independent investigators, such as Robin Ramsay, the maverick editor of the journal Lobster, and of unofficial historians. It will be interesting to compare The Defence of the Realm with the authorised history of MI6, which is set to follow next year.

Susan Williams is a Senior Fellow at the Institute of Commonwealth Studies, University of London. Her latest book is 'Colour Bar' (Penguin)