Binary Trees? Automatically Identifying the Links Between Born-digital Records
I recently presented a summary of a paper I am looking to publish in the Australian Society of Archivists Journal, Archives and Manuscripts, at their annual conference, their theme, Forging Links.
The paper asks how Archives can take small steps toward better archival description of born-digital records, but the techniques discussed also have implications for sentencing, discovery by the end user, and digital preservation.
The abstract reads as follows:
The sheer volume of records that government organisations, and thus government archives, work with on a daily basis means that there is a chance that relationships between individual records will not easily be captured and recorded. This paper begins by suggesting that the relationships described in archival catalogs will remain at the highest levels of abstraction unless relationships can be extracted using automated methods. Relationships that can be generated automatically are described in this paper but will likely be less canonical than archivists are traditionally used to working with. For example a so-called ‘fuzzy matching’ technique is discussed that may reveal the ‘points’ similarity between two records. Extensible databases will be needed to store new links; flexible interfaces will be required to display them. This paper discusses some of the techniques that may currently be available for automatically identifying links between born-digital records by looking at what can be found in the data stream and the relationships digital formats inherently describe. The mechanisms described may be useful for sentencing as well as cataloging and description. While one size will not fit all, some collections may benefit. The paper concludes by discussing briefly what this will eventually mean to the end-user.
It was a useful experience presenting to an audience of talented archivists, and it gave me a lot to think about for improving the final draft of my paper before submission. Submitted a few weeks ago, it is hoped that those improvements will help to see it through to publication. We’ll see how those results look in the coming months.
Before then, the slides summarizing the paper can be found on SlideShare.
And a video of the presentation is available on YouTube.
Feedback and comments are appreciated below.
Check out their website for more information on the Australian Society of Archivists.
Interesting presentation thanks Ross. More automated processing of digital records to enable better discovery is so needed . Haven’t made it through the whole thing yet, but it looks like you may not be aware of this tool I led the development of at Archives NZ: http://openpreservation.org/blog/2012/02/01/dependency-discovery-tool-office-files-code-published/
“The “Office Dependency Discovery Tool” searches through binary office files (.doc, .xls and .ppt) and tries to find any documents or files that are linked to the document.”
http://openpreservation.org/blog/2012/02/01/dependency-discovery-tool-office-files-code-published/
Cheers,
Euan
Interesting. Looking forward to read the improved paper.
Twitter: @redundanton
This fuzzy matching approach sounds genuinely useful for handling the scale problem archives face—curious how messy the results get in practice and whether archivists find the automatically generated links actually helpful or just noise to filter through.
It’s a good question, and something I’d love to revisit given the opportunity.
There are two things to tease apart here: (1) managing expectations, and (2) managing results.
On (1), the sensitivity of the tools at the time of writing was quite high, which, from my perspective, meant that there could be a number of false positives. You need to set the threshold for matches quite high to remove noise, e.g., to avoid matching two files just because they are the same file format containing very little data.
From the perspective of the user, a high threshold still provides opportunities to find efficiencies when processing materials.
On (2), then, you have the opportunity to use fuzzy matching for processing, and I think you can create workflows with succinct reporting looking at “high percentage” matches that help with processing. The paper, however, was largely written with the assumption that material was already largely appraised, so there might be very little additional processing. Focusing, therefore, on an end user’s interpretation of an archive—e.g., comparing two largely identical speeches separated by sub-series or some other delineation. In this example, you will have calibrated the fuzzy tools and will choose to display fuzzy matches as potential other links to follow across a digital archive to surface new connections.
Fuzzy matching is just an additional layer of metadata, analogous to an annotation, that gives you what might be a connection between resources. The archivist or researcher uses this to complement their research.
You could imagine replacing and updating this metadata and its links to other resources as technology improves or as new collections are added.
Fuzzy matching for born-digital records is such a practical angle—I’m curious how the automated linking actually holds up when you’ve got, say, thousands of similar government documents all sitting in the same collection.
Like I say, it needs calibrating. You might only be looking at adding 10s of links to thousands but I’d like to see real numbers today.
You can explore perceptual fuzzing as well. I mention Karp’s paper in an updated blog.
I am also exploring content-based fingerprinting in JSONID but this could definitely be extended to fuzzy analysis for textual content as well.