PRONOM’s dustiest records
Tyler’s recent blog post for the PRONOM Hack-a-thon Week 2024 (my previous for this week), brought up an interesting point about two of PRONOM’s oldest outline records, Real Video Clip (fmt/204) and Real Video (x-fmt/277). How did they end up in PRONOM?
NB. because of the complexity of this post, it may be easier to read in original blog form, than on Mastodon here: https://exponentialdecay.co.uk/blog/pronoms-dustiest-records/
Tyler suggests:
I assume PRONOM originally added these based on MIME types available.
I thought I knew the answer, but it prompted a forensic look at the records to see if what I thought I knew aligned with reality!
As a PRONOM maintainer at The National Archives, UK from 2009-2012 I knew a little bit of the history of the system, we see some of that history impact us today, for example, when we look at the number of records that don’t have descriptions or file format signatures, 156 of those records are so-called x-PUIDs. A mechanism in PRONOM that was never meant to make it into the wild for working on file formats internally without polluting the public record. There are 455 x-PUIDs in total. They made it into the wild anyway (before my time) and so they exist as a symbol of PRONOM’s dustiest oldest records.
Even by the time I had started, PRONOM still had a lot of what we started to call outline records. One of the more positive changes we made to the process back in the day was that we would stop creating outline records; instead, we would focus on records that could be tied to signatures. This didn’t necessarily make the records more correctly aligned with reality, but it meant records had utility and file formats identified by DROID could be tied back to something that PRONOM “knew about”. I believe the process is a bit more flexible these days, allowing individuals to contribute information to records that tie them back to information like MIMEtypes and specifications. It’s clearer the format is “real” even if a signature is yet to be developed (and of course there are a large number of data formats that are hard to even represent in traditional PRONOM signatures any more and so they need a record, even if there isn’t a neat concept of a signature for them).
Okay old-man, but what about Tyler’s thesis?
Stellent and PRONOM
I learned sometime in my tenure at The National Archives that PRONOM had been seeded with a lot of the formats listed in a technology called OutsideIn previously owned by Stellent and now owned by Oracle.
I had always had a feeling that that the scope of this list was largely exaggerated by the company selling the software as it is a marketing tool; and if not exaggerated, perhaps, just not as clearly delineated by format than PRONOM, and rather, by Software, regardless of the properties of a given “format”, e.g. WinZip, and PKZip.
Back to the story though, I was also reasonably sure I would find Tyler’s RealVideo formats in the format listing but, I did not!
I downloaded a CSV summarizing the PRONOM records from api.pronom.ffdev.info with:
curl -X 'GET' \
'https://api.pronom.ffdev.info/pronom_summary_csv' \
-H 'accept: application/csv'
I filtered on outline entries and those without signatures only. I went through the entries still remaining and looked for name matches. I did find some name-for-name matches and some that were close, but no RealVideo or RealVideo Clip.
The matches:
| 7-bit ANSI Text | yes |
| 7-bit ASCII Text | yes |
| 8-bit ANSI Text | yes |
| 8-bit ASCII Text | yes |
| EBCDIC-US | yes |
| Framework Database III | yes |
| IBM DisplayWrite Document 2 | yes |
| IBM DisplayWrite Document 3 | yes |
| Micrografx Designer 3.1 | yes |
| Nota Bene Text File | yes |
| Unicode Text File | yes |
The maybes:
| Cascading Style Sheet | maybe |
| Freelance File 1.0-2.1 | maybe |
| MacPaint Graphics | maybe |
| Microsoft Office Binder File for Windows 95 | maybe |
| Microsoft Works Database | maybe |
| Microsoft Works Database for DOS 2.0 | maybe |
| Microsoft Works Database for Windows 3.0 | maybe |
| Microsoft Works Database for Windows 4.0 | maybe |
| Professional Write Text File | maybe |
| WordPerfect for Windows Document 5.2 | maybe |
| XYWrite Document | maybe |
| XYWrite Document III | maybe |
| XYWrite Document III+ | maybe |
11 exact matches! It’s hardly a headline!
I had hoped that if I found more exact matches it would provide some clues to where some of the older PRONOM entries came from. I expected most of the outline records to come from this list, alas, it isn’t nearly as many as anticipated.
I hoped too that going through the list I might get more clues as to formats that could potentially be deprecated in PRONOM.
As it stands, from the OutsideIn list, the only records I would personally recommend for deprecation are:
7-bit ANSI Text 7-bit ASCII Text 8-bit ANSI Text 8-bit ASCII Text EBCDIC-US Unicode Text File
We know enough now to be almost certain that if something that looks like these files arrives in the archive it will present as a standard text file, and that we will need to rely on determining the character encoding using tools such as Richard Lehane’s characterize (see characterize’s README for more background). It is unlikely we will be able to attach a signature to these records, and we know there are a great deal more encodings in the world than need be represented as PRONOM identifiers.
NB. this might be something to formalize in a PRONOM decision making rubric, connected also, to formalizing approaches for XML based signatures.
A bit of a let down, or is it?
Still uncomfortable with so many outline records and little provenance for them, I wanted to find more information about the source of PRONOM data and so I decided to take a different path — I surfed the internet for answers!
Out of the list of outline records I found a few to be overly specific, or slightly weird, i.e. not really things we hear much about day-to-day, some examples:
| ACBM Graphics | Apple Sound | AutoCAD Plot Configuration File 1.0-R13 | AutoCAD Plot Configuration File R14 |
| AutoSketch Drawing | Btrieve Database 5.1 | CorelDraw Pattern | DEC Data Exchange File |
| DEC WPS Plus Document | Dr Halo Bitmap | Generic Library File | HTML Extension File |
| Hewlett Packard AdvanceWrite Text File | Inkwriter/Notetaker Template | Inset Systems Bitmap | Instalit Script |
| Interleaf Document | Microsoft Excel Add-In | Microsoft Excel ODBC Query | Microsoft Excel Toolbar |
| Microsoft Powerpoint Design Template | Microsoft Print File | Microstation CAD Drawing 95 | NAP Metafile |
| Nota Bene Text File | OS/2 Change Control File | Revit External Group | SAP Document |
| SAS Data File | Scanstudio 16-Colour Bitmap | Schedule+ Contacts | Speller Custom Dictionary |
| Unisys (Sperry) System Data File | Wordperfect Secondary File 5.0 | Wordperfect Secondary File 5.1/5.2 | form*Z Project File |
ACBM graphics? Dr Halo Bitmap? Btrieve database, “5.1”? where are the other five?!!
It gave me pause. I didn’t believe these were all formats well-known to folks who created PRONOM, and I know we didn’t have such an advanced digital transfer program at the time that meant agencies were submitting huge variations of formats to PRONOM for future preservation.
I felt they had to come from somewhere, but where?
Enter Filext.com
Because these formats were very specific I found listings on the internet that I knew had to be part of the story. I had immediate luck just looking for combinations of these names, e.g. ACBM Graphics + NAP Metafile.
In particular I found listings on different websites from hobbyists or universities that all looked the same or similar, e.g.
There were definite matches with PRONOM which we will get to, but I started to wonder about the provenance of these extensions.
I kept looking and I found one clue, a header and footer of a file that looked like those above and read as follows:
Copyright © 2002 Computer Knowledge All Rights Reserved This download for personal use only. Do NOT distribute it to others either alone or incorporated into any software without prior permission from Computer Knowledge. Developers who wish to incorporate portions of the list please see the comments at the end of this file.
Developer permissions... . This total file may not be included in any other software or project which presents the data to the public or portions of the public. Any developer who wishes to include up to (but not more than) 2,000 individual entries from this file is free to do so provided certain conditions are met. These are: . 1) Credit must be given to FILExt. If links are available in the developed product then one must also be provided to FILExt as http://filext.com. . Suggested text: "File extension list courtesy of FILExt. For a more extensive list visit http://filext.com." . 2) Once the extensions are chosen for one product by any developer then these same extensions must continue to be used by that developer for any other projects (i.e., you cannot take one set of 2,000 for one project and a different set of 2,000 for another project; it's a total of 2,000). . 3) If links are available in the developed product then any links appearing associated with any of the 2,000 picked extensions must be included in the product. (This covers future plans to include such links in this list.) . When the project is complete please notify FILExt with the specifics at feedback@filext.com. We're always interested in how the list is being used. Thank you.
Filext.com!
And so I asked myself, how long had filext been around?
As it turns out, quite a while! It was forked from a site called cknow around 2002. cknow.com was registered around 1996 and filext.com registered in 2001.
The first appearance of cknow in the internet archive is late 1996: https://web.archive.org/web/19961219035827/http://www.cknow.com/ and Filext early 2001: https://web.archive.org/web/20010522235126/http://www.filext.com/
The sites were founded by Tom Simondi. It looks like he has been responsible for a lot of the 90s and 00s work around demystifying extensions and getting more information to folk about what to do with them.
Could it be the source of the first PRONOM records?
Comparing some of the many other text-based lists I had found with cknow and filext gave me some confidence that there was some shared heritage with the them, and so I asked, could the cknow and filext lists have also seeded PRONOM?
I picked a list close to 2002 (cknow Extensions: 2000) when PRONOM was first started and began to compare entries for exact matches.
| ACBM Graphics | yes |
| AutoCAD Compiled Menu | yes |
| AutoSketch Drawing | yes |
| Btrieve Database 5.1 | yes |
| DataFlex Query Tag Name | yes |
| Deluxe Paint bitmap | yes |
| DesignCAD Drawing | yes |
| Digital Video | yes |
| Dr Halo Bitmap | yes |
| Frame Vector Metafile | yes |
| Framework Database II | yes |
| Framework Database III | yes |
| Framework Database IV | yes |
| Information or Setup File | yes |
| Inset Systems Bitmap | yes |
| InterBase Database | yes |
| Lotus Approach View File | yes |
| Mathematica Notebook | yes |
| Microsoft Excel Add-In | yes |
| Microsoft Excel ODBC Query | yes |
| Microsoft Excel OLAP Query | yes |
| Microsoft Excel OLE DB Query | yes |
| Microsoft Excel Web Query | yes |
| Microsoft FoxPro Library | yes |
| Microsoft Outlook Address Book | yes |
| Microsoft PowerPoint Graphics File | yes |
| Microsoft Powerpoint Add-In | yes |
| Microsoft Visual FoxPro Table | yes |
| Microsoft Works Database | yes |
| Microsoft Works Document | yes |
| Microstation CAD Drawing 95 | yes |
| NAP Metafile | yes |
| Nota Bene Text File | yes |
| OS/2 Change Control File | yes |
| PICS Animation | yes |
| PageMaker Document 3.0 | yes |
| PageMaker Time Stamp File 4.0 | yes |
| Professional Write Text File | yes |
| Quicken Data File | yes |
| RealVideo Clip <– cc. Tyler! | yes |
| Schedule+ Contacts | yes |
| StatGraphics Data File | yes |
| Structured Query Language Data | yes |
| Ventura Publisher Vector Graphics | yes |
| XYWrite Document III | yes |
| XYWrite Document IV | yes |
46 matches!
| Apple Sound | maybe |
| AutoCAD Device-Independent Binary Plotter File | maybe |
| AutoCAD Drawing Template | maybe |
| Cascading Style Sheet | maybe |
| DEC Data Exchange File | maybe |
| DEC WPS Plus Document | maybe |
| Freelance File 1.0-2.1 | maybe |
| Java Servlet Page | maybe |
| Micrografx Designer 3.1 | maybe |
| Microsoft Office Binder File for Windows 95 | maybe |
| Microsoft Office Binder Template for Windows 95 | maybe |
| Microsoft Office Binder Template for Windows 97-2003 | maybe |
| Microsoft Office Binder Wizard for Windows 95 | maybe |
| Microsoft Office Binder Wizard for Windows 97-2003 | maybe |
| Ventura Publisher | maybe |
| XYWrite Document | maybe |
| XYWrite Document III+ | maybe |
17 maybes!
What did we answer?
Okay, 46 exact matches does not the full listing make (although many (now) full-entries may still have been made from these early listings). Filext may have been an important resource for the first PRONOM records, but it’s also likely that PRONOM had other sources of information. For example, for a number of the Microsoft formats with outline records read like export or save-as listings in previous versions of Microsoft software. E.g. Excel:
NB. I wasn’t actively researching this side of things writing this blog, but I can already see some commonalities, especially Unicode Text!
I know we also had a copy of the Dr Dobb’s Essential Books on File Formats CD-ROM in the archive, and so that may also have been an important resource when PRONOM was creating its first records.
I count only two overlaps with the Stellent list, Framework Database III and Nota Bene Text File.
We did, however, find the RealVideo Clip! And I think we found some decent correlation with a resource that looks likely to have been used partially to populate the PRONOM database.
The era of file extensions
- Throughout my research, I found a lot of similar websites. Filext seems to go furthest back and has the greater pedigree, but in the noughties a lot of other sites seemed to appear to try and provide similar information to internet users, a few of note that seemed comprehensive and particularly well presented:
- Endungen.de (Internet Archive circa. 2002) https://web.archive.org/web/20020603193356/http://www.endungen.de/index.php?changelanguage=049
- Dotwhat.net (Internet Archive circa. 2005): https://web.archive.org/web/20050615021635/http://www.dotwhat.net/
- File-extension.net (Internet Archive circa. 2007): https://web.archive.org/web/20070521093707/https://file-extension.net/
- Filesuffix.com (Internet Archive circa. 2006): https://web.archive.org/web/20060615083930/http://www.filesuffix.com/browse/browse-1.html
- Filext backup: https://www.fileext.com/
I am sure we looked at these sites during my time on PRONOM, although with less frequency given the need to reduce outline records and increase the number with actionable information.
NB. I also learned that TrID has been around since 2003! https://web.archive.org/web/20030612031252/http://mark0.ngi.it:80/
Provenance and prior art
It’s not entirely productive to say I wish we had better provenance for PRONOM records back in the day – but I do!
It makes me reflect on the importance of looking outside of our own walls in digital preservation instead of the constant redundancy of reinvention or ownership.
Often as academics, or those with archival views of the world, we can provide a polish and precision to technology as it exists to make it more usable in an archival context.
But cknow has been around so long, and the Unix utility File was created in 1986.
There’s a parallel history here that we should be recognizing and sharing for our next colleagues.
I arrived at TNA in 2009 and learned about File maybe two years later. As a Windows guy at the time, that might not be uncommon, but I do feel it is on me to have known more. I also think it should have been trivial to access the provenance around some of the records in the database at the time, but more than that – as a field, shouldn’t we all know Tom Simondi? What if the same academic rigour of PRONOM and DROID could have been applied to existing tools like File? What if we had expanded our bubble and recognized digital preservation (or the tools for it) is something people have been doing in all but name for the longest time? What if the people working in parallel on these projects and websites were part of the digital preservation inner-circle community today?
I don’t have answers, but I feel there are lessons there for the future. Not reinventing or rebuilding without good reason is important, but even if we build something new and we have been inspired by something else, continuing to recognize and acknowledge prior art is important.
What do you think?
Also, how do we get these people into a room and celebrate their work, and learn more!
What next?
I don’t think I got very far here but I found it interesting, and I hope other readers may as well.
This is meant to be a PRONOM hack-a-thon blog and I don’t know if I have pushed the sticks forward that much but maybe there’s a bit more to reason about in the outline records, for example, around the plain-text formats mentioned above and a few more identified along the way.
| 7-bit ANSI Text | x-fmt/21 | Recommend deprecation |
| 7-bit ASCII Text | x-fmt/22 | Recommend deprecation |
| 8-bit ANSI Text | x-fmt/282 | Recommend deprecation |
| 8-bit ASCII Text | x-fmt/283 | Recommend deprecation |
| Unicode Text File | x-fmt/16 | Recommend deprecation |
| EBCDIC-US | fmt/159 | Recommend deprecation |
| MS-DOS Text File with line breaks | x-fmt/130 | Recommend deprecation |
I noticed in the outline entries some low-hanging fruit that I might focus on next opportunity if someone else doesn’t get there first, these would be:
| Cascading Style Sheet | x-fmt/224 | Consider adding CSS to the record name | A signature should be feasible |
| Document Type Definition | x-fmt/315 | Consider adding DTD to the record name | A signature should be feasible |
| Extensible Stylesheet Language | x-fmt/281 | Consider adding XSL to the record name | A signature should be feasible |
| HTML Extension File | x-fmt/417 | Related to Microsoft’s ISS server | A signature may be possible |
| Standard Generalized Markup Language | x-fmt/195 | Consider adding SGML to the record name | A signature may be possible |
| Still Picture Interchange File Format 2.0 | fmt/113 | Related to JPEG | A signature should be possible |
| Structured Query Language Data | fmt/206 | Consider adding SQL to the record name | A signature may be possible |
| Dreamweaver Lock File | fmt/335 | A system file, there may be an entry in the NSRL database | A signature may be possible |
A little more on the history of extensions websites
The complete filext text file (allext.zip)
It took a few jumps, but I found the complete downloadable text file from Filext.com. I don’t think it exists any more and I don’t think the internet archive managed to grab a copy. Apparently it was quite a chunk of data to download on the web once upon a time, but they eventually found a way to release a zipped text file:
Via one jump we get to the “whole list” page:
https://web.archive.org/web/20020605164206/http://filext.com/wholelist.htm
And then to confirm our absolute interest in downloading it, we get to the a2z file:
https://web.archive.org/web/20020606071418/http://filext.com/a2z.htm
Which would have taken us to the zip file, alas, never captured on the Internet Archive anyway, maybe it is on other Memento compatible servers:
https://web.archive.org/web/20060117000000*/http://www.filext.com:80/allext.zip
Keeping filext up to date
Filext still asks for registry data to help keep it up to date. That’s pretty cool!
https://filext.com/faq/gather_data_for_filext.html
1 │ Echo OFF
2 │ CLS
3 │ assoc > filext_submission_output.txt
4 │ Echo ---------- >> filext_submission_output.txt
5 │ ftype >> filext_submission_output.txt
6 │ Echo Thank you. The output file has been created and
7 │ Echo named filext_submission_output.txt and it should
8 │ Echo be in the same place where you saved this batch
9 │ Echo file. All that is left now is to send that file
10 │ Echo to FILExt. Attach it to an E-mail sent to the
11 │ Echo address: filetype@filext.com
12 │ Echo The E-mail subject should be: Submission
13 │ Echo Thank you.
14 │ Pause
15 │ Exit
Filext as a source of learning
The filext faqs and community seemed particularly helpful and interesting back in the day:
https://web.archive.org/web/20090322040812/http://filext.com/faq/
File extension aggregator
The file-extension.net website started an aggregator project around 2007 and it’s still running today!
http://file-extension.net/seeker/
Some bonus images…
As I was working on this, I found irony in Google Sheets glitching, I managed to grab some screenshots along the way. Thanks for reading everyone!
@exponentialdecay shout out to @Thorsted for inspiring me to finally go through some of this in a bit more detail.
@exponentialdecay Google sheets glitching is wild!
@joshuatj @exponentialdecay yeah! First time it happened I got excited as ever but couldn't capture it, but then it kept happening, so must be a *thing* on that combo of hardware, in this case a Lenovo Chromebook.
@exponentialdecay very cool and detailed investigation. There are a bunch of Macintosh based format I need to go through and make recommendations for. Some will never have signatures because of their structure. Others are too vague. “Macintosh Text File”, really? They must have some from one of these sources you mention.
@Thorsted @exponentialdecay 💯 I guess it's not a lot of "noise" for those of us working on it regularly, and I feel like we mostly add to PRONOM when we have something new these days, but every little bit helps.