Digital Archiving Archives - ross spencer :: exponentialdecay.digipres

Image shows two layered waveforms, one a corrupt waveform and the other a good original. The corrupt form is in red and the uncorrupt one is green.

Revisiting bsdiff as a tool for digital preservation

I introduced bsdiff in a blog in 2014. bsdiff compares the differences between two files, e.g. broken_file_a and corrected_file_b and creates a patch that can be applied to broken_file_a to generate a byte-for-byte match for corrected_file_b.

On the face of it, in an archive, we probably only care about corrected_file_2 and so why would we care about a technology that patches a broken file?

In all of the use-cases we can imagine the primary reasons are cost savings and removing redundancy in file storage or transmission of digital information. In one very special case we can record the difference between broken_file_a and corrected_file_b and give users a totally objective method of recreating corrected_file_b from broken_file_a providing 100% verifiable proof of the migration pathway taken between the two files.

Crayola's 1997 Techno Brite crayon set with color names created to market the Crayola website, including names featured here such as World Wide Web Yellow, Point and Click Green, and Cyber Space Orange

Looking after your URLs: tikalinkextract eight years on

We might not have a second life, but what if I told you there was a second internet? Not the deep web, but another web that we engage with nearly every day?

Think about it, that QR code you scanned for more information? That payment link you followed on your electricity bill? The website you’re told to visit at the end of a television ad?

The antipodes of the internet are these terminal endpoints, material and not necessarily material objects that represent the end of the freely navigable web — the QR code on a concert poster is the web printed onto the physical world. There is every chance it will be scanned and followed by someone from a mobile device, but it’s a transient object, something that will exist for a short amount of time, and then disappear into the palimpsest of the poster board or wall it was pasted on until it eventually disappears.

This is part of the materiality of the internet that has long fascinated me. Perhaps it comes from being a student of material culture, but if we look around, we see the Internet everywhere!

Animated GIF of various corrupted frames from Johan's Y2K cover.

The sensitivity index: Corrupting Y2K

In December I asked “What will you bitflip today?” Not long after, Johan’s (@bitsgalore) Digtial Dark Age Crew released its long lost hidden single Y2K — well, I couldn’t resist corrupting it.

Fixity is an interesting property enabled by digital technologies. Checksums allow us to demonstrate mathematically that a file has not been changed. An often cited definition of fixity is:

Fixity, in the preservation sense, means the assurance that a digital file has remained unchanged, i.e. fixed — Bailey (2014)

It’s very much linked to the concept of integrity. A UNESCO definition of which:

The state of being whole, uncorrupted and free of unauthorized and undocumented changes.

Integrity is massively important at this time in history. It gives us the guarantees we need that digital objects we work with aren’t harboring their own sinister secrets in the form of malware and other potentially damaging payloads.

These values are contingent on bit-level preservation, the field of digital preservation largely assumes this; that we will be able to look after our content without losing information. As feasible as this may be these days, what happens if we lose some information? Where does authenticity come into play?

Through corrupting Y2K, I took time to reflect on integrity versus authenticity, as well as create some interesting glitched outputs. I also uncovered what may be the first audio that reveals what the Millennium Bug itself may have sounded like! Keen to hear it? Read on to find out more.

René Magritte's The Lovers, Paris 1928 (Photographed at MoMA, NYC in 2017

Unrealized ideas: Unintentional Secrecy in the Era of Openness

Tyler recently posted this quote:

“History unprocessed is opportunity unrealized”

It reminds me of an unrealized article I wasn’t able to get written and into the wild, but it’s an important thought I would like to share nonetheless.

Proposed for James Lowry’s ACARM Symposium in 2015, I wanted to discuss when government is unable to adequately fund day-to-day effort, and research and development in the archive sector, leading to inefficient and potentially ineffective processing pipelines for records of archival value accessioned from government agencies and commissions.

It was just an abstract, but maybe folks have thoughts about this? Have we moved on since the early to mid 2010’s? What modern metrics do we have available to us today to see the progress? What does the advent of the new US administration mean for issues like this? As well as increasing worldwide authoritarianism?

Cat's Meow from the Offner Dynograph EEG

What will you bitflip today?

I want to let you into a secret: I enjoy corruption. Corrupting digital objects leads to undefined behavior (C++’s definition is fun). And flipping bits in objects can tell us something both about the fragility, and robustness of our digital files and the applications that work with them.

I had a pull-request for bitflip accepted the other day. Bitflip is by Antoine Grondin and is a simple utility for flipping bits in digital files. I wrote in my COPTR entry for it that it reminds me of shotGun by Manfred Thaller. The utility is exceptionally easy to use (and of course update and maintain written in Golang) and has some nice features for flipping individual bits or a uniform percentage of bits across a digital file.

My pull-request was a simple one updating Goreleaser and its GitHub workflow to provide binaries for Windows and FreeBSD. I only needed to use Windows for a short amount of time thankfully, but it’s an environment I believe is prevalent for a lot of digital preservationists in corporate IT environments.

Bitflip is a useful utility to improve your testing of digital preservation systems, or simply for outreach, but let’s have a quick look at it in action.

Client-side file format identification and reporting pipeline with Siegfried and Demystify Lite

With thanks to the sponsorship of Archives New Zealand and Richard Lehane for his great coding expertise and his collaboration; Demystify Lite has a new feature — Siegfried!!

Richard recently posted about this work on LinkedIn but lets look at this effort in more detail below.

Published: PREMIS Events Through an Event-sourced Lens

Not long after my first Code4Lib article I had another idea to run by the team there, and elected to see if my paper looking at events in the PREMIS metadata standard would be of interest to them and the readership.

My paper PREMIS Events Through an Event-sourced Lens was published April this year.

I take a look at the content of this paper below and plug a few gaps that I have been thinking about since its publication.

Digital Preservation as a Thought Experiment

Back in 2017, I had an abstract accepted for a chapter in the ALCTS Monograph: Digital Preservation in Libraries: Preparing for a Sustainable Future. With my author’s copy now available, I take a look at the background and its genesis below. The complete monograph is a fascinating read with some great contributors. You can find it online at the ALA Store.

Digital Preservation Risk analysis, Uncertainty, and Digital File Format Obsolescence (European Conference on Digital Archives (2010))

Full paper: The application of risk models to digital file format obsolescence I have seen a lot of conversation about…

Digital Preservation: The Job Application

I recently, applied for a pretty cool role in digital archiving and digital preservation. Applying for the role and considering…

Tag: Digital Archiving

Revisiting bsdiff as a tool for digital preservation

Looking after your URLs: tikalinkextract eight years on

The sensitivity index: Corrupting Y2K

Unrealized ideas: Unintentional Secrecy in the Era of Openness

What will you bitflip today?

Client-side file format identification and reporting pipeline with Siegfried and Demystify Lite

Published: PREMIS Events Through an Event-sourced Lens

Digital Preservation as a Thought Experiment

Digital Preservation Risk analysis, Uncertainty, and Digital File Format Obsolescence (European Conference on Digital Archives (2010))

Digital Preservation: The Job Application

Follow ross spencer :: exponentialdecay.digipres :: blog