{"id":2380,"date":"2025-03-19T09:44:02","date_gmt":"2025-03-19T09:44:02","guid":{"rendered":"https:\/\/exponentialdecay.co.uk\/blog\/?p=2380"},"modified":"2025-11-24T18:07:21","modified_gmt":"2025-11-24T18:07:21","slug":"file-format-building-blocks-primitives-in-digital-preservation","status":"publish","type":"post","link":"https:\/\/exponentialdecay.co.uk\/blog\/file-format-building-blocks-primitives-in-digital-preservation\/","title":{"rendered":"File format building blocks: primitives in digital preservation"},"content":{"rendered":"

A primitive in software development can be described as:<\/span><\/p>\n

a fundamental data type or code that can be used to build more complex software programs or interfaces.<\/span><\/p><\/blockquote>\n

\u2013 via <\/span>https:\/\/www.capterra.com\/glossary\/primitive\/<\/span><\/a> (also Wiki: language primitives<\/a>)<\/span><\/p>\n

Like bricks and mortar in the building industry, or oil and acrylic for a painter, a primitive helps a software developer to create increasingly more complex software, from your shell scripts, to entire digital preservation systems.<\/span><\/p>\n

Primitives also help us to create file formats, as we\u2019ve seen with the Eyeglass example I have presented previously<\/a>, the file format is at its most fundamental level a representation of a data structure as a binary stream, that can be read out of the data structure onto disk, and likewise from disk to a data structure from code.<\/span><\/p>\n

For the file format developer we have at our disposal all of the primitives that the software developer has, and like them, we also have \u201cfile formats\u201d (as we tend to understand them in digital preservation terms)<\/em> that serve as our primitives as well.\u00a0<\/span><\/p>\n

<\/p>\n

Take for example plain-text (x-fmt\/111<\/a>) \u2013 if I write this blog in plain-text we have a document, or record, something recognizable as the written language, literally, it’s a blog. Yet, if I use this same foundational \u201cfile format\u201d and instead write:<\/span><\/p>\n

#! \/usr\/bin\/bash\r\nset -eux\r\necho \u201chello world\u201d\r\n<\/pre>\n
I have a shell script that’s going to be evaluated as at least three commands in a Linux environment.<\/p>\n
If I am asked what are the significant properties of either my blog, or the shell-script, through the lens of plain-text it would make very little sense.\u00a0<\/span><\/p>\n
Asked however, what the properties of the thing that is the blog are important (the record), or the properties of the shell script that I need to case about then, we would have a very different conversation.<\/span><\/p>\n
In the second instance I haven\u2019t a simple record or document any longer, I have something approaching code, and with that, an exponentially larger number of dependencies, including what platform it might run on, what commands might be invoked, and something that when run, could succeed or fail depending on different conditions.<\/span><\/p>\n
The first take away here is that plain-text is a file format primitive, it helps us to build more complex formats.<\/span><\/p>\n
We can take this further, like in something approaching research data in XML, or JSON to pick two examples.<\/span><\/p>\n
These \u201cformats\u201d, to take a very high-level viewpoint, are built upon plain-text. In PRONOM we have the classification “text” and \u201ctext (structured)\u201d (structured text) and this is an important delineation, structured text means that we have a format that is likely to have very specific instructions about how to process the data that is encoded.<\/span><\/p>\n
XML for example is a textual representation of a tree structure, and we talk about its elements in terms of roots, nodes, namespaces, parents, and children.<\/span><\/p>\n
JSON encodes, platform agnostically, the language primitives, of programming languages, (ECMA-404<\/a>, RFC7159<\/a>) – slices\/arrays, keys, values, dictionaries, and so on.<\/span><\/p>\n
XML and JSON both have important properties, both rely on being well-formed and valid, for example, to be functional, e.g. piping JSON in to JQ<\/a> will allow it to be queried – but if the JSON is missing a single curtly-bracket, it can\u2019t really be accessed or queried at all – at least not without finding the problems with the file and fixing them, e.g. through the lens of JSONLint<\/a>.<\/span><\/p>\n
Access and query are interesting terms here \u2013 what does accessing and querying something like XML or JSON mean?<\/span><\/p>\n
Well, it depends.<\/span><\/p>\n
If we extend the concept of a file format primitive to structured text such as XML and JSON, we can begin to understand that we don\u2019t simply “write XML”<\/a> for the sake of writing XML. We write XML alongside its own semantics either implied by some code or programmer’s purpose, or intention; or written explicitly, through the use of a schema – this adds the property of validation to XML’s already very specific ideas of being well-formed and valid<\/em>.<\/span><\/p>\n
The point is to access and query the XML means understanding its structure and what it encodes, and how it encodes it.<\/p>\n
Likewise, when we write JSON it is usually to be conformant to some sort of specification as well; again, implied, say, through code, or explicit through a technology like JSON schema<\/a>.<\/span><\/p>\n
Stepping back for a second, our technology stack is growing, we have a plain-text file, on which we have Unicode code points for each character in an XML or JSON document, and all of the rules associated with those languages, and alongside those, we have have another plain-text file schema document, encoded in its own format, with its own rules, and describing the rules of the document we have written. It\u2019s getting complex.<\/span><\/p>\n
\"Image<\/a><\/p>\n
Given this view we can see that preservation of the XML can’t be done in isolation, but, perhaps less clearly, understanding the XML can’t be understood without context.<\/p>\n
But what context?<\/p>\n
If I give you a PRONOM record you need to understand PRONOM’s XML schema. If I give you a METS document you have to understand the METS schema. Both documents have very different semantics that are needed for the preservation and understanding of either as records.<\/span><\/p>\n
Lets briefly look at the Eyeglass format again. This time, here are three JSON representations of the data:<\/span><\/p>\n
Representation 1: An array of tuples and two strings<\/strong><\/p>\n
[\r\n    [\r\n        \"-3.35\",\r\n        \"+0.50\"\r\n    ],\r\n    [\r\n        \"-0.25\",\r\n        \"-1.00\"\r\n    ],\r\n    [\r\n        130,\r\n        80\r\n    ],\r\n    [\r\n        0,\r\n        0\r\n    ],\r\n    [\r\n        0,\r\n        0\r\n    ],\r\n    [\r\n        0.66,\r\n        0.5\r\n    ],\r\n    [\r\n        12,\r\n        12\r\n    ],\r\n    \"Distance and Close Work.\",\r\n    \"Patient's eyesight needs correction. History of diabetes in family but indicators found. Standard checkup interval recommended.\"\r\n]<\/pre>\n
Representation 2: One long array<\/strong><\/p>\n
[\r\n    \"-3.35\",\r\n    \"+0.50\",\r\n    \"-0.25\",\r\n    \"-1.00\",\r\n    130,\r\n    80,\r\n    0,\r\n    0,\r\n    0,\r\n    0,\r\n    0.66,\r\n    0.5,\r\n    12,\r\n    12,\r\n    \"Distance and Close Work.\",\r\n    \"Patient's eyesight needs correction. History of diabetes in family but indicators found. Standard checkup interval recommended.\"\r\n]\r\n<\/pre>\n
Representation 3: A collection of name-value pairs<\/strong><\/p>\n
{\r\n    \"sphere\": [\r\n        \"-3.35\",\r\n        \"+0.50\"\r\n    ],\r\n    \"cylinder\": [\r\n        \"-0.25\",\r\n        \"-1.00\"\r\n    ],\r\n    \"axis\": [\r\n        130,\r\n        80\r\n    ],\r\n    \"prism\": [\r\n        0,\r\n        0\r\n    ],\r\n    \"base\": [\r\n        0,\r\n        0\r\n    ],\r\n    \"distance_acuity\": [\r\n        0.66,\r\n        0.5\r\n    ],\r\n    \"near_acuity\": [\r\n        12,\r\n        12\r\n    ],\r\n    \"purpose\": \"Distance and Close Work.\",\r\n    \"observations\": \"Patient's eyesight needs correction. History of diabetes in family but indicators found. Standard checkup interval recommended.\"\r\n}\r\n<\/pre>\n
At a human level, “representation 3” might be the easiest to understand as the “keys” are self-describing; we just need to understand when a value is for the left-eye or right-eye, and we might also want to understand the language of the text, and its encoding. Both representations “1” and “2” are terse in different ways and it is probably best to assume that the rules for decoding these will be in someone’s code for the Eyeglass format somewhere. <\/span><\/p>\n
All three representations encode exactly<\/em> the same information and they\u2019re not the only ways of encoding it, they are just the first three I landed on. Using JSON as a building block for the Eyeglass format we can be more or less expressive as desired, we can even add more complexity, like Base64<\/a> for text, we can even sign, or encrypt values (if patient data is sensitive, e.g. like the encryption used in SOPS<\/a>).<\/span><\/p>\n
The question is, which version will I receive at my research data archive if we haven\u2019t reasonable standards around design and documentation of research data? And does it matter if I haven\u2019t a schema to work with that will allow me to reuse any of this information?<\/span><\/p>\n
The answer is no, it doesn\u2019t matter. At a human level it is very likely that I haven\u2019t anywhere near enough information to decode what is here without the background information.<\/span><\/p>\n
This is why a holistic, records based analysis of digital objects is important in digital preservation. You can\u2019t simply take one object out of context and on merit and understand what preservation means. <\/span><\/p>\n
With the JSON above, for example, in the context of research data it is far more important for me to know, what the data was, what its schema was, whether the output was repeatable as part of another script, i.e. for the recreation and verification of results. Even understanding if the data could be replayed byte-for-byte is an incredible property, if present. I might have one version of the JSON on disk, and then another I can output in code, and if they align bite-wise, i.e. their checksums match, I have a very strong link between the research project and its outputs (we’re getting close to those diplomatics<\/a> again!)<\/em>.<\/span><\/p>\n
When asked to analyse a dataset recently, most of the above was missing in the collecting methodology and documentation. These are bigger problems than the preservation of the data itself.<\/span><\/p>\n
When we wrote Re-imagining the Format Model. Introducing the work of the NSLA Digital Preservation Technical Registry<\/a> (and iPRES 2014<\/a>) we spoke about four structures that we can use to understand file formats:<\/span><\/p>\n