The first in what I hope will be, a series of blogs. This blog has been torn from an introduction to another I was writing and so started off one way, and has ended up this way. Comments and thoughts appreciated below.
I started some new code yesterday in my new chosen language Golang. I guess I have been dabbling in it just over a year, with my most significant release so far (development of a Key Value Access Language) being released just under a month ago.
Writing yesterday, I was thinking about my current area of work in digital preservation in an archive and thinking about the time and space we (the community) find ourselves in where the paradigm is still shifting from paper to digital.
The shift calls for many new skills, but the one that receives quite a lot of attention is that of computer programming. Does the modern archivist or librarian need to also be a programmer to be able to do their job?
The short answer for me is no. There are too many other skills required, and too many other challenges to be working on. Archival science is the most important, but across the board, other skills may include, leadership, scholarship, communication, and empathy. There also seems to be a continual need for outreach in our discipline, upwards, sideways, internally, externally – convincing others of the importance of an archive, why we need an archive, and the importance of information, and how to use it.
So I am not sure when you would find the time to learn programming! That being said, digital literacy is, without a doubt, pretty important. I believe that there are a handful of skills at the top of the digital literacy list that can be ticked off pretty easily for most with just a bit of time and dedication – they are probably the most important – checksums, command line skills, basic digital preservation and data transfer tooling, etc. I can talk about this in another blog sometime.
What about programming? My gut feeling as I write is that it gets more important that you have people with the ability to program in your ranks the larger your organization gets. This might sound counterintuitive because you might say that the smaller organizations need to do more with less. I am not sure I see it this way. I believe it is important for larger organizations like my own to lead the community by developing tools and skills, and sharing those with the widest possible audiences, especially within their own jurisdiction.
If smaller institutions take on this work as well, then it’s also important for us to promote that work on the same stage, so when I describe my view, I would like to make it’s clear it’s one of pragmatism not one driven by corporate narcissism. Our work, if possible, is better served by being more open, and evenly spread, across the sector.
The primary reason I think it is important In larger organizations, is that there are other IT requirements for a greater number of users, from human resources through to accounting. The easy thing for any company to do would be to make as many of the requirements of any department as generic as possible.
This may or may not be a good thing. For me, what is most important is that if your team has special requirements (and the teams directly involved with the archives and preservation should!), you have someone capable of talking the right language, someone who can also keep IT bods and management in check when jargon enters the frame, and someone who can take requirements up the chain, and back down again, clearly and effectively, and without losing their impact.
Without programmers in your ranks then I believe it is possible for your organization to become inert through the homogenization of tech skills, thus not letting you develop the capability you need, pushing the envelope, and not developing the capabilities smaller institutions need to follow your lead on.
My experience? My skill extends back to my time before archives. I learned by developing tooling across the breadth of the computing stack.
When we talk about the stack we talk about programming (a stack of technology) upwards from the hardware, to the network interfaces, up to the server, application, database, and user interface levels. So everything from code that works as infrastructure for other code, to code that a user will interact with on their desktop.
There are key aspects of my job that only require a certain amount of digital literacy. I slipped into this for a while at The National Archives, UK, (TNA) as I wrestled with the ways in which a government organization might work.
I arrived as a programmer, but for nearly a year, I had little to do that was truly technical. It would have been easy (but not enjoyable) to slip into a world of report writing and endless meetings. During this time it was even difficult for me to be granted the time to write file format signatures for PRONOM; not a good situation to be in.
The change in my work came when I realized that I couldn’t let the work required to talk to the community, and also maintain PRONOM and DROID slip, but at the same time recognizing that I still needed to be part of the report writing and meeting processes (I guess bureaucracy has a place?).
As such, I started to write tooling that would support my work and let me get more done.
One of the first programs I created for TNA was a tool to generate sets of random digital files for load testing of various processes, including ingest of Submission Ingest Packages (SIPs). I created other tools to handle exceptions with JPEG2000 files, and a JPEG2000 validation tool, just before the much more fully featured jpylyzer came onto the scene and we switched to that.
I then created the PRONOM Signature Development Utility, and, as my time at TNA was coming to an end, one of my more significant works, the Skeleton Test Suite Generator. A tool for creating file format signatures, and one to test DROID’s ability to adhere to a signature’s self-describing specification.
It is the story of the Signature Development Utility that is most important here – It was written to convince the powers that be that it was a tool that needed to be written.
The goal of it was to let folks outside of TNA develop signatures and submit them to the PRONOM team for testing and inclusion in the database. It was created to take the load off our team at the time, and to democratize and demystify the signature development process. It did also have the potential to get more people doing this work within the organization as well.
I left TNA in early 2012 and so I only saw an initial use of the tool by Georgia Tech Research Institute for a collaborative project. I didn’t see its full acceptance, or not, within TNA, but, I am more than happy seeing how it has been used externally in the time following. It has more than warranted the time and effort taken to do the work of my own back.
Since then, and for often the same reasons, I have continued to program as much as possible in my work, both at home, and at Archives New Zealand.
Every project is open source, but not every project will be useful or applicable to others, but that’s not necessarily the important part. The important part continues to be the journey. Every snippet of code that is written that we can use in some way gives me more understanding of the realities of our requirements.
Every piece of code is like a new hypothesis. Repeated use is a demonstration of the proof of its importance, less frequent use demonstrates the need to tweak a hypothesis or forget about it altogether.
And every piece of code enables me to develop understanding, and share that with my team. I will talk, and demonstrate, and blog. Now when we interact with other technology providers the conversation goes two ways.
I benefit from other coders too, notably, recently, my colleague Andrea who has just left for pastures new. I also benefit from non-coders, the archivists I work for. It is important that all folks are supported and a feedback loop is created. My work will often only be as good as my received understanding.
So this blog has been largely incidental in its genesis. It started as an introduction to another blog I will write soon about developing programming skills but became too unwieldy that way. As such this writing might not be a neat package of ideas as might be hoped, but I’ll endeavor to conclude as best as possible.
Do you need to become a archivist programmer?
No. But it might be fun nonetheless! 😉
There are a lot of digital literacy skills I do not think require programming. Knowing how to create checksums, analyze data, run tools, develop file format signatures, understand file format specifications – these are all things we can learn without learning programming.
It is pretty great what we can do actually, and while I talk about programming helping me to demystify conversations I think that probably only applies in the rarer, lowest level conversations that we may sometimes have. Digital literacy in most of what I describe above will help regardless.
What programming may be able to do for you is:
- Better put you in a position to translate ‘impossible’ from a solutions provider to ‘possible but will take some effort’
- Give you additional confidence to call folk up on jargon and high priest mentality
- Give you an understanding of what is reasonable to expect from solutions providers across the board in terms of quality, testing, development paths, etc.
- Allow you to develop prototypes to demonstrate value to managers
- Allow you to develop better software requirements
- Better enable you to understand and demystify your tools
- Better enable you to understand and demystify your inputs
- Allow you to plug the gaps in your work flows, or create tools better suited to your work flows
- Enable you to lead, and up-skill your organization and community by sharing the output of your endeavors
- Allow you to have a bit of fun looking at your content laterally
These benefits do by no means make programming a requirement. As I note above, a feedback loop is important, and I feel there are more important skills: archival science, leadership, scholarship, communication, and empathy; that need to exist to benefit our work best.
Good programmers are scholars in their own right. Good programmers listen to other’s and they listen to your requirements. Good programmers have skills applicable to any industry. There is only so much our brains can pack in, in any line of work, and so I’m not sure if it is even feasible to just absorb programming alongside other years of schooling. Your passion alone has to drive it.
If one can avoid Dunning-Kruger; then on top of other digital literacy skills, a little can go a long way, which I think might be the important thing to take away.
Other suggested reading on this blog: The Civil Servant Perspective