Author: Ross Spencer

Digital preservation domain expert and full-stack software developer. A researcher, analyst, and programmer. I am currently a senior software architect for Iona Systems Ltd. focused on the Web3 Orcfax (Cardano) Oracle, and Arkly (Arweave) projects. I am a freelance digital preservation developer and advisor. I worked previously on the Archivematica and PRONOM, and as a digital preservation specialist at Archives New Zealand, and The National Archives, UK. I have 14 years of experience working across the sector and have a keen interest in information and records management.

Image of the foundations of a new building being erected in Wellington New Zealand, circa 2017.

File format building blocks: primitives in digital preservation

A primitive in software development can be described as:

a fundamental data type or code that can be used to build more complex software programs or interfaces.

– via https://www.capterra.com/glossary/primitive/ (also Wiki: language primitives)

Like bricks and mortar in the building industry, or oil and acrylic for a painter, a primitive helps a software developer to create increasingly more complex software, from your shell scripts, to entire digital preservation systems.

Primitives also help us to create file formats, as we’ve seen with the Eyeglass example I have presented previously, the file format is at its most fundamental level a representation of a data structure as a binary stream, that can be read out of the data structure onto disk, and likewise from disk to a data structure from code.

For the file format developer we have at our disposal all of the primitives that the software developer has, and like them, we also have “file formats” (as we tend to understand them in digital preservation terms) that serve as our primitives as well.

Image of a children's learn to code textbook by Usborne Books. The page shows a snippet from a computer game in BASIC called "Escape". The illustration is of three menacing looking Cyborgs.

Informed consent: considering steganographic techniques to fingerprint Generative AI output

Artificial intelligence (AI) is a polarizing topic. For every reasoned assessment of the technology and its potential to make some of our smaller, onerous, or more repetitive tasks easier, there are probably 100 reactive pieces predicting some radical overhaul of societal norms, from the service industry receiving new intakes of out of work software developers to laypeople taking on roles traditionally occupied by those of a college education, if they just start “asking their AI the right questions“ ¯\_(ツ)_/¯

The amount of AI-propaganda is draining, and the reaction is often spread across the board too, some cheer leading, some decrying, plenty taking their time to offer skilled and nuanced rebuttals, or suggestions for improvements.

I find myself largely trying to stay out of the conversations. A lot like blockchain conversations 8 years ago, it will take another half decade for the hype-cycle to plateau for us to see where it can truly complement our work.

One part of the conversation that is increasingly harder to ignore, is being informed about when AI has been used in the generation of text or images. It is the property of knowing, or having the tools to know is what I feel is the most important.

How can we be better informed about when AI is used, so that we are better prepared as consumers, to receive and understand content?

In this blog I want to explore the potential for steganography techniques to be used in the output of AI to fingerprint content and provide a way for front-end mechanisms to identify it, as we might file formats using magic numbers, so that users can be given the chance of informed consent: the opportunity to opt-in or out of whether we engage with AI content or not.

Animated GIF of various corrupted frames from Johan's Y2K cover.

The sensitivity index: Corrupting Y2K

In December I asked “What will you bitflip today?” Not long after, Johan’s (@bitsgalore) Digtial Dark Age Crew released its long lost hidden single Y2K — well, I couldn’t resist corrupting it.

Fixity is an interesting property enabled by digital technologies. Checksums allow us to demonstrate mathematically that a file has not been changed. An often cited definition of fixity is:

Fixity, in the preservation sense, means the assurance that a digital file has remained unchanged, i.e. fixed — Bailey (2014)

It’s very much linked to the concept of integrity. A UNESCO definition of which:

The state of being whole, uncorrupted and free of unauthorized and undocumented changes.

Integrity is massively important at this time in history. It gives us the guarantees we need that digital objects we work with aren’t harboring their own sinister secrets in the form of malware and other potentially damaging payloads.

These values are contingent on bit-level preservation, the field of digital preservation largely assumes this; that we will be able to look after our content without losing information. As feasible as this may be these days, what happens if we lose some information? Where does authenticity come into play?

Through corrupting Y2K, I took time to reflect on integrity versus authenticity, as well as create some interesting glitched outputs. I also uncovered what may be the first audio that reveals what the Millennium Bug itself may have sounded like! Keen to hear it? Read on to find out more.

"Bei der Buche", a landscape architectural installation by landscape architect and photographer Karina Raeck. Created in 1993 in the Wartberg area north-east of Stuttgart.

wikidata + mediawiki = wikidata + provenance == wikiprov

Today I want to showcase a Wikidata proof of concept that I developed as part of my work integrating Siegfried and Wikidata.

That work is wikiprov a utility to augment Wikidata results in JSON with the Wikidata revision history.

For siegfried it means that we can showcase the source of the results being returned by an identification without having to go directly back to Wikidata, this might mean more exposure for individuals contributing to Wikidata. We also provide access to a standard permalink where records contributing to a format identification are fixed at their last edit. Because Wikidata is more mutable than a resource like PRONOM this gives us the best chance of understanding differences in results if we are comparing siegfried+Wikidata results side-by-side.

I am interested to hear your thoughts on the results of the work. Lets go into more detail below.

René Magritte's The Lovers, Paris 1928 (Photographed at MoMA, NYC in 2017

Unrealized ideas: Unintentional Secrecy in the Era of Openness

Tyler recently posted this quote:

“History unprocessed is opportunity unrealized”

It reminds me of an unrealized article I wasn’t able to get written and into the wild, but it’s an important thought I would like to share nonetheless.

Proposed for James Lowry’s ACARM Symposium in 2015, I wanted to discuss when government is unable to adequately fund day-to-day effort, and research and development in the archive sector, leading to inefficient and potentially ineffective processing pipelines for records of archival value accessioned from government agencies and commissions.

It was just an abstract, but maybe folks have thoughts about this? Have we moved on since the early to mid 2010’s? What modern metrics do we have available to us today to see the progress? What does the advent of the new US administration mean for issues like this? As well as increasing worldwide authoritarianism?

A poem by Kay Ryan - An Elephant in the Room The room is almost all elephant. Almost none of it isn't. Pretty much solid elephant. So there's no room to talk about it.

Interviewing in digital preservation: a duty of care and community

Sometime in 2024, I received zero feedback for a job interview—one of at least five interviews without any feedback in the last eight years.

The thing is, digital preservation is very niche. Those five roles probably represent a good number of institutions actually hiring specialists and likely represent some of the best chances for jobs in the future.

Not getting a role is part and parcel of interviewing, but in not providing feedback, a didactic moment was lost—a moment of community connection and outreach—and simply an act of care.

Furthermore, loops are not closed, processes feel incomplete, and of course, you will likely know the person who gets the role ahead of you. Trying to measure yourself against that individual will likely be in the back of your mind when you next meet or work with these individuals because you have been left questioning by the recruiter.

And before it is suggested that this is just a ‘you’ thing—let’s say conservatively, five people interviewed for each of the five positions I applied for. Assuming everyone is treated equally, that’s 20 people missing out on something critical to improving their skill set, interview technique, or helping them find more suitable jobs in the future. I guarantee, you ALL deserve feedback. It is also 20 people that each recruiter has missed an active opportunity to build a stronger bond with, who will sing the praises of the process and the organization; this is important.

Cat's Meow from the Offner Dynograph EEG

What will you bitflip today?

I want to let you into a secret: I enjoy corruption. Corrupting digital objects leads to undefined behavior (C++’s definition is fun). And flipping bits in objects can tell us something both about the fragility, and robustness of our digital files and the applications that work with them.

I had a pull-request for bitflip accepted the other day. Bitflip is by Antoine Grondin and is a simple utility for flipping bits in digital files. I wrote in my COPTR entry for it that it reminds me of shotGun by Manfred Thaller. The utility is exceptionally easy to use (and of course update and maintain written in Golang) and has some nice features for flipping individual bits or a uniform percentage of bits across a digital file.

My pull-request was a simple one updating Goreleaser and its GitHub workflow to provide binaries for Windows and FreeBSD. I only needed to use Windows for a short amount of time thankfully, but it’s an environment I believe is prevalent for a lot of digital preservationists in corporate IT environments.

Bitflip is a useful utility to improve your testing of digital preservation systems, or simply for outreach, but let’s have a quick look at it in action.

Tyler's Halloween Matryoshka Dolls represent the internal complexities of container file formats. The dolls here have formats attached to them representing different ways they might be nested, with ZIP and OLE2 being the primary containers that can be handled in DROID and Siegfried at present.

A year in file formats 2024

A great write up from Francesca at TNA about the past year for PRONOM via Georgia at the OPF.

It’s great to see the continuing work including vital translation of guides into other languages. Francesca includes a couple of shout outs to some pieces I have contributed in my spare time this year; including a collaborative workshop with Francesca, David, and Tyler at iPRES2024.

What we could do with a second life…

Cleaning up some posts today for clarity or for improving their appearance in ActivityPub instances I didn’t want to lose this quote introduced to us at Archives New Zealand in a visit from Verne Harris back in 2017. It represents the need for a second life to apply all of the lessons learned in this one – in the GLAM sector, everything we learn getting up to speed, to learn how to work within our institutional boundaries, to align with corporate strategy, or just to hustle to have our work recognized and valued.

My colleague Andrea references the quote a lot and I am often reaching to recall it.

We work in the dark – we do what we can – we give what we have. Our doubt is our passion, and our passion is our task. The rest is the madness of art. – Henry James, The Middle Years

A tree at sunset photographed from the train on the Bodendsee in Southern Germany

Versioning as memory?

So, it turns out my theme of the moment is code hygiene (or maybe memory?).

Today I am thinking about versioning, especially in relation to its impact on digital preservation; both software preservation and the impact of versions on long-term preservation efforts in other contexts.

Follow ross spencer :: exponentialdecay.digipres :: blog