Winding down at Orcfax: a retrospective
With the recent announcement that Orcfax is heading into operational mode, it’s a bittersweet moment that means our adventure in the web3 space is almost at an end.
Operational mode means that we are finally able to leave the systems that we have created running and publishing content all-day-every-day with minimal monitoring. Our salaries are paused and we won’t receive another paycheck unless something drastically changes in the next three months.
At the end of July, everything will be switched off, and the team will move onto its next adventures.
But what was Orcfax? What was my role there? What happens next?
Orcfax
There was, and still is a lot of negativity around web3 and this is largely because it became synonymous with “crypto” and “crypto-currency”. I tend to disambiguate web3 as a series of technologies, more generally distributed technologies, distributed ledgers, cryptography, multiple signature signing (multi-sig), and programmable currency. These alone, are important innovations that may or may not all live into the next three decades like the foundation of the world-wide web, but they have taught us a lot, and they are technologies we should follow in the interest of, among other things, self-sovereignity, provenance, security, and authenticity.
Another disambiguation that helps is to remove the term “web3” entirely and look at it as the decentralized web (DWeb) which has much more positive connotations with technologies enabling peer-to-peer (p2p), local-first, resilient networks — check out the range of topics in the call for papers at this years DWeb Camp for more.
“Crypto” on the other hand can happily disappear with the rampant tokenization of ALL-THE-THINGS ultimately causing the “industry” to collapse in on itself. Yes, we need methods to incentivize pro-social behavior in collaborative DWeb networks but it needn’t be all about the trading of imaginary money.
There are also a lot of dark and nasty corners in the crypto world too, and we will be much better off when it finally goes away.
How was Orcfax good?
Our CEO, Peter Van Garderen, whom I worked with on Archivematica and who founded Artefactual and later Orcfax, recognized that blockchain technology had strong potential. He also acknowledged that one of its biggest weaknesses was something easily solvable but also easily exploitable if not addressed carefully.
The weakness is that a blockchain has no connection to the outside world, as it is not an “Internet” in the classical sense. Data must be put “on-chain” by a service called an oracle, but to do this well, the data must be trustworthy and authentic.
Seeing a potential market, Orcfax’s oracle solution sought to make this possible by using archival principles to collect data from heterogeneous sources along with its para- and meta-data and package this information so that it could later be audited. Each subsequent record that we created would reinforce the archival bond allowing data-points to be traced over time to help auditors identify incorrect or manipulated data.
We aimed to become good actors in blockchain and sought to improve the ethics and quality of data in blockchain technologies as we progressed.
An on-ramp to archives and digital preservation on the DWeb
Orcfax was about plugging an immediate and crucial gap in the blockchain. We hoped that the desire for authentic and trustworthy data would allow the company to grow and provide enough security to enable more research into other DWeb principles and, eventually, enable the team to start developing new archival and digital preservation tools using those.
The project started strongly with us designing the archival packages using schema.org vocabulary and Bagit. We built a new service called Arkly. Arkly was built on top of a layer zero (L0) blockchain called Arweave that promises long-term (permanent) storage.
Our Web and JavaScript guru Gabriel built a standalone viewer for the archives which began to realize an ideal Peter and I have often discussed about standalone, rich archival information packages (AIPs). I am sure this is something we will return to.
We hoped more oracle use-cases would appear which would allow us fund our development further but unfortunately none appeared, and even our focus on trading which still fuels too much of the conversation on blockchain proved unfruitful as many companies could not afford prohibitive costs built into blockchain pricing, and many did not commit to funding decentralizing efforts.
Were the project to continue we would have continued to apply archival and digital preservation principles to our work. We are also sure that Web3/Dweb primitives, already briefly discussed, would have fueled our research in the world of archives and digital preservation.
There are still many interesting strands to follow around decentralization, pro-social engagement, incentivized/collaborative archiving and digital preservation platforms, self-sovereign-identity, permissionless registration of records, chain of custody, long-running records of provenance, and much more.
Senior system’s architect
It has been about four years since Peter reached out and asked about my availability to work on Arkly, and then Orcfax. Much of that time I have been Senior System’s Architect for the project. While Peter was CTO my role was focused on piecing everything together.
At times I would describe myself as an interpreter between different stakeholders, at other times my head was deep down in the weeds of understanding requirements and minimum viable products and how to deliver those on time.
In spite, or because of Conway’s Law we ended up creating over 100 GitHub repositories, half of those open to the public. Each of those repositories contribute to our currently federated solution, running on three networks (production, pre-production, and preview), on 20 servers across the globe. Repositories range from methods of exploring data (largely created by Gabriel) to on-chain scripting (Dominic), schemas and documentation and research (Christian, Peter, and the rest of the team), and collectors, validators, devops tooling, deployment scripts (George and myself).
I am proud of all of it, but especially being able to help focus and drive delivery. I had to take on board a lot of understanding of many different concepts and programming languages and styles. I had to give up a lot of detailed control on aspects I wasn’t an expert on, to learn how to access the information and tools I needed to bring pieces together with the team to make our network available.
My timesheet tells me that I worked 4,952 hours (across 45 months) since I began working for Orcfax and in that time I have worked in Python, Golang, Aiken. I have touched upon JavaScript. I have learned how to keep servers up and running 24/7 and how to write decentralized long-running processes communicating across HTTP and Web Sockets. I have written screeds of unit and integration tests and performed endless hours of user testing. I have written code review guidelines and I have code reviewed and refactored (and have been code reviewed and refactored!) I have taken meetings with 100s of people across the sector and have performed roles as client and contractor. At one point we managed to get our new permissionless network up and running (a crucial part of the next phases of our work) for just over six months with over 50 participants from across the globe. This was yet another long-running process and we collected over 8 million data points during that time.
I appreciate all the participants from the permissionless network and the time and resources they committed. It also taught me a lot about communication with those outside of our organization and a lot about writing new technology off the cuff to patch our own flaws as we experimented.
I am definitely also forgetting a lot more about what I learned during the last 45 months, but I will continue to reflect and build on what I have gained.
An ambition realized
Back when I was still working a support desk for a risk management software company, I watched the lead architect John Burgess do his thing, and I thought to myself, this is what I want to do when I finally get back into programming, and finally get the opportunity.
A series of events happened after that, from quitting a year-long stay in Australia after four-days to learn C++, to my first professional role as a developer, all of my time in digital preservation, to working at Artefactual, and then finally landing this position on Peter’s team. Thank you Peter!
It’s not the end of my ambitions. I still want to commit to a role like this, on a digital preservation team (or related) and I hope I will get the opportunity to do so without some of the time pressures of working in a startup, with more time to consciously develop my soft- and hard-skills.
Peter’s recent recommendation on LinkedIn helps me to see I probably did an okay job, but I always keep a running list of things to improve that I want to work on.
Ross is a rare talent with deep domain knowledge in both digital preservation and best-practice software engineering. His many years of experience ranges from working in national archival institutions, to maintaining open-source projects, to Web3 projects like those we tackled with Cardano and Arweave. While working together on the Orcfax oracle project, Ross served as Senior Systems Architect, providing the technical direction and software component backbone for our decentralized data feed and archiving architecture. He is a practical, empathetic and professional technical lead that provides clear direction and feedback. I would strongly recommend him as a valuable addition to any software development project, particularly those in the digital preservation and Web3 space.
The team
The “we” in everything I have said up to this point is the team. We managed to achieve a lot together. The team was made up of Christian, Dominic, George, Gabriel, and Karsten. A cross-domain team, with many of us newly learning the mentioned technologies to accomplish our work.
Everyone brought something different but complementary to the table. Christian, for example, has incredible attention to detail, which was important for translating our efforts into technical schema or documentation. Gabriel’s web experience gave us rich interfaces and new ways of engaging with our choice of blockchains and our own data. George was a great partner in all of the Python we delivered and developed new and innovative tools for extracting data from external sources. Dominic (or Waalge or Alge) is an incredibly talented computer scientist and functional programmer, and he also kept us all on our toes when it came to precise semantics. Last but not least, Karsten is a wizard of communication, always able to distill our technical efforts into posts for blog articles or social media and always ready to be the extrovert when communicating with our community.
During the periods of time we had most freedom to discuss our work and plan for the future we engaged in a lot of lively and engaging debate with a lot of disagreement turning into consensus as each of us put forward our ideas or found our way. It was a lot of fun and a big learning experience.
Thank you to each of you, and I hope I didn’t get on your nerves, too much.
The future
If our deadline is reached, then we will take great pride in what we developed but I imagine a few of us will be happy to put some of this work on hold for a while and go back to the domains we’re most comfortable with, albeit, bringing our new skills and experiences with us.
I am concerned about the future; from low salaries in the field to a realization that we may have devalued the skills of software engineering in the GLAM sector — this latter point is something I am still processing having interviewed for three roles emphasizing the need to build material software, but two out of the three interviews did not even make time to ask about software development processes and what that means and what we need to bring to the table to make robust, resilient, and maintainable software today.
We have a lot of people writing code in the field, we need that, but like an archivist or librarian trains in their discipline, a software engineer trains in theirs, and at some point you hope a 25 year career will make an impact to some of the large, e.g. EU-scale projects looking at delivering future facing outputs.
Goals
I have some goals for the future. I want to continue to look at the DWeb technologies I have discussed from peer-to-peer technologies to the cryptographic principles I want to work with to enable better ways of protecting people’s identities and provide better guarantees of authority and authenticity. I think this will be useful in government.
Peter has recently published a core set of ISO-based digital records requirements which is another strand I will follow up on.
Most will know I haven’t really slept on digital preservation while working at Orcfax.
I delivered a PARBICA report on implementing AtoM in the Pacific with my colleagues at Archives New Zealand.
I have also been involved with workshops at iPRES in Belgium and New Zealand, and delivered three talks at No Time to Wait.
I have also been working on GOCFL for the University of Basel, and delivering a RO-CRATE packaging tool FHNW Basel as well.
I want to continue my work there looking at JSONID and perhaps a few other ideas I have bubbling away.
One thing I am still musing about is the difference between community and collective and think the latter is under-researched in the GLAM sector. Figuring out a way to do more with the latter and make enough money for a team to subsist could lead to some new and interesting opportunities.
One potential effort might be a PhD if I can find the right topic. My hope is that I can continue to innovate in a more formal setting and potentially gain some time to look at other new threads for enabling digital preservation across communities and disciplines if there’s a mentor out there who is willing to take me on. There is also a question of funding, but we’ll see what is possible.
A brief pause…
That’s me for now, and for this 4-year effort. We keep our fingers crossed for something happening between now and July and our new CEO Christian has been working his heart out for a while now trying to make that so. Most of us want to keep continuing to contribute to the Orcfax effort, even if our hours are reduced while we work on other projects.
I’m going to be organizing a lot of my ideas and thoughts and writing. This will be my first piece of writing since we stopped. Hopefully some of it was interesting to you as you read? Let me know if it triggers any thoughts! But thank you for reading if you did. Like I said, the experience of wrapping up is bitter sweet; we are able to get another three months of hope because our effort was successful, we just haven’t been successful enough. The pause also gives most of us our first moment to stop in all that time, and yeah, it’s a lot (of mostly positive experiences) to process. Thanks once again 🫡👋🎤.
Further reading
Example archival packages
Orcfax packages demonstrating para- and meta-data can be found on GitHub.
Orcfax glossary and documentation
It may seem small, but I think we’re all proud of the small glossary of terms we created for the project, as well as the documentation we emphasized at all points so that the project objectives were clear and could be understood by everyone.
GitHub organizations for Arkly and Orcfax
Orcfax on Medium
