Preserving our digital past

FOIMan highlights the important work of UK pioneers to preserve digital records for future generations.

Thank goodness for Vint Cerf. Cerf’s up because he has been speaking at a conference in the US about the dangers of a “forgotten century”. He is highlighting the problem of digital preservation which is an important one. And because he’s a “web pioneer” and a Google Vice-President, the media are listening to him.

If you were to read the BBC News website or the Guardian this weekend, you’d be forgiven for thinking that this is a problem that has just dawned on very clever people in the US. But the truth is that whilst it is welcome that this issue has finally attracted the attention of journalists, archivists and people in the IT industry have not only been aware of these issues for some time, but have also been putting forward solutions. Many of them are right here in the UK.

In a nutshell, the problem is twofold. One, the hardware that runs computer programs is regularly superseded. On my records management courses, I illustrate this by producing a 7″ floppy disk from the 1980s. The hardware that can read that disk only exists now in a handful of museums, but when I was at school, it was used there and in most offices.

Two, the software that is used to create the programs, to manage your email, to retain your photographs, to write letters – also changes all the time. Each time you get a message saying that a new version is available the chances of being able to open a document created in the original version are reduced. Software manufacturers are focussed on creating something that will do lots of new sexy things rather than something that will continue to open your dissertation from ten years ago.

20 years ago this was a problem in the pharmaceutical industry. Research increasingly depended upon technology that produced data which could not be recorded or preserved by traditional methods. This was a concern because regulatory approval for drugs required the experimental data to be available for long periods. And if you wanted to demonstrate that you had discovered a drug for valuable patent purposes then again you needed the records to prove it. A drug like Viagra, say, discovered by scientists in the UK working for Pfizer, is worth billions to the company. So the records proving its discovery are also worth billions.

Back then I was starting my career in records management, and one of the reasons for pursuing it was that at Pfizer I saw it at the cutting edge. Not only did my colleagues invite experts in the field like David Bearman to visit us to discuss the problems, but they were proposing solutions too. They developed, with the help of a UK company called Tessella, a system called the Central Electronic Archive, specifically to retain – and preserve – this important experimental data. The CEA has since been retired, but the work done in establishing it helped to ensure that its contents remained accessible and could be migrated to its successor systems.

This work started in the pharmaceutical industry but its benefits can now be felt in the public sector. My former boss, David Ryan, was headhunted from Pfizer to set up the National Archives’ Digital Preservation Unit. The Unit has produced useful tools such as PRONOM, a database that maps the compatibility of different versions of software so that organisations can work out how to open documents created in older versions of the software. It has also established a programme to extract digital data from central government departments. One example was a 3D reconstruction of a shipping disaster used at an inquest which otherwise could never have been captured and preserved. In establishing the technical infrastructure for these services, the National Archives has continued to rely on Tessella, who won the Queen’s Award for Enterprise in 2011 for their work on this.

The Digital Preservation Unit’s next head, Adrian Brown, has subsequently gone on to establish a digital preservation programme in the UK Parliament, and is widely recognised as an expert in preserving digital formats. Adrian has recently written a handbook on digital preservation to help others looking to ensure their records will be available for decades to come.

There is still a long way to go here, and Vint Cerf is absolutely right to highlight the issue. But much good work has already been done around the world, including here in the UK where pioneers in industry and in the public sector have shown the way.