As Scoop hits Netflix, it’s time to open the secret files on Prince Andrew

Documents are the lifeblood of historians, the tools with which we build our picture of the past. For most historians and biographers there is a surfeit of sources to draw from, ranging from documents in public archives and private collections to accounts in books and interviews.

For the royal historian, however, there is very little beyond often inaccurate press cuttings, and briefings by “sources” in royal circles. Even Netflix reenactments of infamous interviews only allow an interpreation of what the royals themselves chose to put in the public domain.

This is because there remains a deference and culture of secrecy with regard to the royal family. They are largely exempt from the Freedom of Information Act; the Royal Archives have no public inventory – rather like a restaurant with no menu; what papers are deposited in the National Archives are subject to a series of exemptions; and those who have worked with them are almost inevitably subject to confidentiality contracts.

Writing about the royals operates on a lobby system with favoured journalists fed titbits, tame writers rewarded with quangos and those who choose to operate outside the cosy arrangement marginalised.

We have the absurd situation of publicly or covertly authorised biographies of royal figures, of members of the royal family appearing on American chat shows talking about royal life, where Prince Harry can write about the most intimate aspects of royal life from a few months ago for commercial gain, and royal households constantly briefing against each other, yet historians cannot see files which are 100 years old. This is the mark of a banana republic and not a mature democracy.

The revelation from one of my freedom of information (FOI) requests that royal papers are not made public until 105 years after the birth of a particular, is a disservice to history. It means historians cannot look at anything to do with Queen Elizabeth II, even before her accession to the throne, until 2031 and of personal interest, as I am writing his biography, the papers of the Duke of York are closed until 2065.

This is troubling because this restriction applies to his role as trade envoy – a public appointment that was underwritten at great expense by the taxpayer and about which there are many unanswered questions. Who accompanied him on these delegations drumming up trade for Britain?

It is rumoured that David Rowland and his son Jonathan, who supposedly paid off some of the Duchess of York’s debts, were on some of these official trips, including to the World Economic Forum on the Middle East in Sharm el Sheikh in 2008. But we are not being told because both the Foreign Office and the Department of International Trade cannot “identify” any files relating to the trips made.

When pressed with internal reviews and the intervention of the information commissioner, they will trot out details of a few factory visits but continue to hide behind a series of FOI exemptions, not least section 37: communications with the sovereign. A request to the Ministry of Defence for an innocuous-sounding file on his parachute training in 1978 was refused on the grounds of law enforcement, national security and health and safety.

It is a problem I repeatedly encountered writing my biographies of Lord Mountbatten and the Duke of Windsor. The papers relating to Mountbatten’s murder over 40 years ago remain closed because it is claimed it is still “an active investigation”, even though the bomb maker was convicted, served a sentence and then released under the Good Friday Agreement in 1998.

Last year I requested a 1932 police protection file relating to the then Prince of Wales. Dozens of similar files have been available at the National Archives for 20 years. They contain useful titbits on his movements but nothing remotely secret. Yet 90 years later, the Metropolitan Police refused to release the file on the grounds that it would jeopardise the present safety of the royal family.

That decision was upheld by the Information Commissioner’s Office, so I took the matter to a tribunal. A judge asked if I would supply examples of information from other protection files of the period but, when I sought to do so, I discovered that the 20 files I had highlighted in my submission – and which had been publicly available for more than 20 years and much copied by historians – had been mysteriously withdrawn from the National Archives.

As the former MP Norman Baker, author of And What Do You Do? What the Royal Family Don’t Want You to Know, has said about royal archives: “There’s no reason for these to be kept secret. The normal excuse given is that it’s to uphold the dignity of the crown. But the dignity of the crown is upheld by them not behaving in an undignified manner.”

With a new reign, it is now time for more transparency with regard to royal records, not least with a public inventory of the papers held in the Royal Archives. There also need to be parliamentary enquiries into the current failure of public authorities to adhere to FOI legislation, most notably by the Cabinet Office, and more robust action taken by the regulator of information rights, the ICO, to those public authorities who break the law.

If our history is to be written accurately, we should have to have the majority of records made available – not just those a government department or the royal household believes we should see.