The data the NHS has collected is key to the future of healthcare

Data saves lives. Since 1948, the NHS has collected detailed records and data, on tens of millions of patients, from the vast population of the UK. GP records – as just one example – contain detailed information on every diagnosis, prescription and test result, for every patient in the country.

This data – because of its detail, breadth of coverage, and the ethnic diversity of the UK population – has almost unfathomable power to to save lives, both in Britain and around the world. It can be used to research pioneering treatments, discover which treatments work best, learn more about side effects, monitor and improve quality and safety across NHS services, drive research and innovation across healthcare, and much more.

The UK’s response to the pandemic has demonstrated this power in a breathtakingly short space of time: data has been pivotal in understanding who is most at risk from Covid-19, identifying new treatments, monitoring vaccine safety and effectiveness, and much more. Now, data is helping the NHS tackle the post-pandemic backlog.

But data does not do this work on its own. Data must be shaped, checked, and curated into shape. It must be analysed. It must be communicated, and acted upon. And throughout all this, data needs to be housed and managed securely, because NHS records contain the deepest medical secrets of every citizen in the country.

This is no small ask, and there have been many false starts. Previous attempts to extract and disseminate large volumes of NHS patient records have resulted in millions opting out of their data being used in research and analysis at all.

Against this backdrop, the government commissioned us to conduct a detailed review on how to deliver better, broader, and safer use of NHS data in research and analysis. Our report, published in April this year, made 185 wide-ranging recommendations, but the core themes are tangible and accessible to all.

Firstly, the NHS must move away from sending copies of millions of patients records to analysts at a vast number of diverse destinations, and instead consolidate all data analysis into a very small number of highly secure Trusted Research Environments. On this model, the analyst comes to the data, and leaves only with their results.

These platforms actively obstruct data misuse by rare bad actors, and keep detailed transparent logs, to earn trust from patients, professionals, and privacy campaigners. They reduce the need for burdensome information governance processes, and allow us to expand the number of people working on NHS data.

Secure platforms also bring huge efficiency gains, because they reduce duplication of effort on common tasks like data cleaning and curation. Lastly, they are already the longstanding norm in many other sectors, such as for those working on census data.

Secondly – with a more technical hat – we must promote the use of modern, open methods for analysis and research, ensuring that all code and scripts written for data curation and analysis are shared openly, with appropriate technical documentation, to all other data users.

Thirdly, we must take the dispersed community of NHS analysts and build it into a structured, modern analytic profession with training, opportunities for career progression, clear job descriptions tied to technical skills, and realistic salaries where expensive specific skills are needed.

Lastly, and most technically, we must face up to the hard problem of data curation, which is the key to interoperability and efficient analytics. Specifically we must recognise curation as a complex, standalone, high status technical challenge of its own, and meet it with devoted teams, shared working practices, shared code, shared tools, and shared documentation.

Two months after we published these recommendations, the government published its new NHS Data Strategy “Data Saves Lives”.

The strategy commits to implementing secure data environments as the default across the NHS, and states that – as public services are built with public money – the code they are based on should be made available across the health and care system.

Read more from our series on ‘How to heal the NHS’ by clicking here

Now the challenge lies in delivery. Technical challenges are hard, but there are ways to increase the chances of success.

Government and the NHS must recruit domain experts – with deep relevant technical skills – into senior leadership positions on technical projects, to sit alongside the admirable generalists we already have in these roles.

Alongside this, we must build impatiently, but incrementally: avoid panic purchase of black box services, and instead identify teams, datasets, and regions that can act as pioneers for secure platforms and modern, open working.

If we resource a handful of teams well, with instructions to deliver – but also to build capacity and templates for all to follow in the second wave – then we can deliver a generational shift on better use of NHS data, in just two years.

Every new analyst will arrive to find the data they need, checked, curated, documented, and ready to run, in secure platforms, with active positive support from the public and professionals.

This will reap huge rewards across the global research community. It will drive rapid innovation across the whole life sciences sector. It will help drive change across the NHS, where smart use of data can help monitor and rapidly improve the quality, safety and cost effectiveness of all care, for all patients, across the country.

And it can all be done for less than the cost of digitising one hospital.

Ben Goldacre is a clinical researcher at the University of Oxford where he is director of the Bennett Institute for Applied Data Science, and Bennett Professor of evidence-based medicine in the Nuffield Department of Primary Care Health SciencesJessica Morley is a social science researcher at the University of Oxford where she is the policy lead for the Bennett Institute, and a Wellcome-funded DPhil candidate at the Oxford Internet Institute