ross spencer :: exponentialdecay.digipres :: blog
Digital preservation analyst, researcher, and software developer
With thanks to the sponsorship of Archives New Zealand and Richard Lehane for his great coding expertise and his collaboration; Demystify Lite has a new feature — Siegfried!!
Richard recently posted about this work on LinkedIn but lets look at this effort in more detail below.
In my post from 2012: Genesis of a File Format, I created a new file format – the Eyeglass file format. The format provides a mechanism to persist information about a patient’s eye health following a checkup at an opticians. Today in 2023 we can use the format to understand how to make use of Kaitai Structs for understanding file formats.
Given the disclaimer that I am not actually an optician and that the format is purely illustrative, let’s look at the eyeglass again below.
I have been working on a Python template repository as part of my day-job at Orcfax.
It is based on the popular pypa sample project and adds important tooling that supports the quality assurance of projects that many developers are expected to engage with.
In my template repository I add editor defaults, linting, and prepare the repository for unit tests, and then deployment.
I have migrated a copy of the template I created for Orcfax to a new file format organisation I have created to capture work I am doing around tools such as ffdev.info (the PRONOM signature development utility).
The new template repository can be found here: ffdev-info/template.py.
I want to talk about how this tooling can be used as a way of understanding legacy, or new code that you are going to be looking at. Looking at how linting can be useful for learning and understanding.
Following the previous posts, bringing this all together meant three different applications.
paintergoblin.py – creates the images, can be run standalonewikigoblin.py – retrieves data to tweet from the Wikidata SPARQL servicestwittergoblin.py – tweets for us! Either a random Wikidata image or from am existing Wikidata linkWe create Tweetable information using the wikigoblin. We perform the Tweet using twittergoblin. In between the paintergoblin has to create his art!
We’ve seen examples of the images from the original zine.
How do we turn this concept into something real, and automated?
One thing that held the Painter Goblin project back was finding a data source to get images from.
There are potentially hundreds of sources out there, but! The path of least resistance means that:
** A hackable URI is a URI pattern that can be cycled through using computational techniques, even if the underlying data isn’t entirely well-known. E,g, http://example.com/image/0001, http://example.com/image/0002, for subsequent pages, for lack of a more concrete example.
I wanted to explore heritage sources such as Europeana, TROVE, DPLA. I struggled to search these effectively though, and struggled to see how I might automate using them. I recognise they have APIs. I’ll revisit them in the future as I look to expand the Painter Goblin’s corpus.
Enter Wikidata.
Continuing the story of the Painter Goblin and following part one, the idea of a Twitter bot started to form earlier this year.
In part 2 I take a very brief look at what is needed to write a bot and get it publishing.
Learn more below.