As always, the National Archives has revealed nothing of substance – the truly juicy stuff will always remain covered up

The strange thing about the annual release of “secret” information from the National Archives is how much of it doesn’t deserve to be kept secret. It is usually, by turns, unsurprising – the sort of gossip you’d be likely to see covered by any national news organisation – and no real threat to anyone’s idea of the national interest. It is, in other words, an annual reminder of why freedom of information is so vital, because the authorities’ default setting is to lock everything up until everyone affected is dead – even the trivia.

The truly juicy stuff, as we must all realise, always remains covered up, virtually indefinitely, if it hasn’t been destroyed already. There is a sort of perpetual official secrecy machine, operating across political parties and across generations – and it is harming us.

Take spam fritters. I have a vague recollection about these being part of the national commemorations for the 50th anniversary of the D-Day landings in 1994, and them being mocked.

With its characteristically inept touch, someone in the John Major government thought it might be a bright idea to revive these treats from the era of food rationing as a way to mark the ultimate sacrifice paid by those on the bloodied Normandy beaches in June 1944.

In other words it looked like a stupid stunt at the time, and it’s no shock that Dame Vera Lynn made her distaste clear to all concerned.

Then we find out that, as prime minister, Margaret Thatcher didn’t have such a high opinion of Nelson Mandela as his status as a secular god should have implied. On a phone call to the newly released human rights superhero in 1990, Maggie found he had a bit of a “closed mind”. This, however, is what she thought of anyone who didn’t agree with her, who wasn’t “one of us” in the phrase of the time.

Given that she had spent the last decade resisting pressure to place tough sanctions on apartheid South Africa, and never once appeared on stage at a Red Wedge rally in a “Free Nelson Mandela” T-shirt, I can’t say I’m astonished she didn’t hit it off with Madiba. Besides, world leaders are allowed not to like each other or, in Donald Trump’s case, anyone.

So again, no surprises there, and you wonder what trivia is currently being concealed from us – let alone, much more seriously, some of the big stuff about Brexit. After all, the May government has had to be forced to release the Brexit economic damage assessments, and it was almost toppled over preventing the rest of us from reading the relevant legal advice about the withdrawal agreement. And who knew, come to think of it, about the Windrush scandals, or the “hostile environment” regime operated by Theresa May at the Home Office? Again no one fell off their chair when they found out about any of these, and it was right that the public found out about them – but why were they secret at all?

You get the impression that we still only know a fraction of what we ought to know about Brexit. Even the legal advice released looks like an incomplete version of things.

Meantime, all sorts of historical stuff remains deeply secret, and will probably remain so for decades to come. Apparently there are four thick ledgers of material about Jack the Ripper (from 1888) that remain classified. We will probably never know the truth about Edward VIII’s political leanings and sympathies for Hitler and Germany. The same goes for the handling of the Falklands War, nuclear testing in the 1950s, various famous British spy scandals or even aspects of “European policy” in the 1990s. On a slightly different note, the private diaries of the late Kenneth Williams for 1965 are so incendiary that the British Library, which now effectively owns them, won’t let anyone read them. They must be good.

I doubt, when the time comes, that there will be much enlightening information dished out about the Iraq invasion of 2003. Indeed, the Blair government was notorious for informal, unminuted “sofa” government, and using Post-it Notes to evade scrutiny, even though it passed the Freedom of Information Act. Cynical.

Much the same goes for the millions of deleted emails in recent years by governments and public agencies, such as the Bank of England and Financial Conduct Authority – on the financial crisis, bank bailouts, the recession, the coalition of 2010 and much else. These records are probably lost forever, their secrets not even resting in the archives at all.

Ironically, when data storage, retrieval and search has never been easier, it is also easier to keep and destroy secrets. Back in 1956, the prime minister Anthony Eden ordered the cabinet secretary Norman Brook to collect all paper records of collusion with France and Israel in the Suez affair and destroy them, and he duly did. Today’s deletions of embarrassing and damaging material are quicker and no less effective and final. All the public will ever get is spam fritters.