Programming things: Giving up… (or at least getting bitten by semver and Golang’s unforgiving nature, and wanting to!)

There are good days, and there are bad days when coding, and you never stop learning. Today was not a good day, although in the build up to it, I really thought things were going well.

A little bit of maintenance

One of the packages I have been working on for Siegfried is spargo. spargo is a Golang based SPARQL query library (SPARql-GO, get it?!) and it worked well enough for the initial release of Siegfried 1.9.x which was significant for introducing the Wikidata identifier.

That being said, for the upcoming releases of Siegfried I wanted to make sure there was enough test-coverage in this package as well as the other new packages that I am introducing (more on that another time). Reviewing spargo I found three main issues:

• Only one unit test in total testing only one scenario.
• Lazy error-handling, calling panic() explicitly for any error I decided to handle this way.
• Logging in a package, where logging should be determined by the caller of the package based on feedback from the library, e.g. receive an error, then log it.

I had a bonus issue which was an itch I really wanted to scratch at some-point and that could be characterized as enabling JSON to be serialized from the code’s data structures, instead of storing a separate copy of the same data in the same structure <– see, always learning!

Finally, as my work continued I spotted problems with the releases on https://pkg.go.dev/ and it was not possible for anyone to download specific tags from the code if only legacy functionality was needed. Only “commit” hashes could be specified, see Siegfried’s use here.

github.com/ross-spencer/spargo v0.0.0-20200323024642-38971d4365a7 <-- Just a v0.0.0, timestamp, and a hash-part.

There is a way to tag releases and make them more friendly for the Golang ecosystem, so I set about fixing ALL THE THINGS!

Wait, I also wanted to be a good open source citizen and so I set about it with the following semver (semantic versioning) strategy:

• v0.0.2 would be tagged (initial release from March 2020 and landed in Siegfried)
• v0.0.3 would be minted and tagged (fixes to Go module system)

These two versions were massively important to me. They represented two versions I know would work with Siegfried as is. Since it was released and in the wild, I didn’t want to mess with that at all.

• v1.0.0 would be created adding tests and fixing the logging issue by returning errors to the caller.

Version 1.0.0. represented a component I was comfortable with. An older “versions” have been live in Siegfried since November and I have been using it for my new developments without issue. By adding new tests and removing logging I was making it “stable” (in my opinion) as well as changing the interface, which is usually why the major digit is incremented in semver. In the worst case scenario, Siegfried would need to use the 0.x.x. versions (if for some reason I couldn’t get the code there updated). In the best-case scenario I knew the interface change would need only a minor tweak to Siegfried to drop-in, and so would maybe be a good option.

• v2.0.0 would be created, removing the excess data from the code’s data structures. Again changing the interface and so upping the major version.

This version was designed to be future facing and didn’t need to land in Siegfried at all until some-point in the future.

So, I bit the bullet, thought to myself, it’s time to be bold about versioning, and set out on the road to victory.

Okay but…

Cut to the next day and the packages are not showing on pkg.go.dev:

No 2.0.0 to select. No 0.x.x versions. What is worse, the 1.0.0 version was showing as “latest” and it really was only the 1.0.0. version.

What happened? 

Golang has a convention in its standard library that apparently extends to third-party libraries. I didn’t know. There is more information on the subtlety of this here.

Semantic versioning but not as we’d like it Jim.

Or maybe it is… It’s just that when you look into what Golang is asking for here, you find out what this person (89z) found out here about the increased maintenance overhead. And do please observe their workaround (because my magic eight-ball ? is calling me about my future…).

Golang cannot delete the rogue versions and there is no period of grace. They can only delete the entire package and ask that you create a new library with an entirely different path, e.g. I could use exponential-decay/spargo.

The convention means my Github tags do not align with Golang’s. Users of my package are going to get two different experiences. It is pretty messy. And I don’t feel happy that I have lost the ability to change it either. I feel pretty stupid and embarrassed about it; that I have opened myself for criticism, and yeah, that’s not a great feeling.

So what Ross? Stop whining!

So, I chatted to Richard (the Siegfried maintainer), and they pointed me at the retract directive which is only just available in Golang 1.16. It’s pretty similar to what 89z did above, but using the infrastructure a little more.

So I’ll find my way back to something “less of a mess”.

What I wanted to express was that before I reached out. I was pretty close to creating an entirely new package and going through the tagging process again (note: I’ve already been through it once since the issue arose and have completely messed up my own versioning to try and satisfy pkg.go.dev). I was also pretty close to deleting the original package, but this seems like it has the potential to mess up Siegfried at some point. So it doesn’t look like a great idea.

That is to say, despite my best efforts; literally, over the course of 8.5 hours, very deliberately, following through on everything I’ve learned in my work, I was pretty much at a loss. And, now, with a workaround in sight, I’m not feeling any better about the situation. And it brings back a lot of memories of inflexible or unforgiving aspects of technology, and y’all might find yourself facing it at times.

I feel like giving up would have been just fine here. But at some point I’ll be glad I’ve persisted, and I’ll be able to apply my learnings here to the next package I write (it’s not far from being released, so…).

And this “trauma” will be a distant memory. But in the moment, yeah, it sucks. So I feel you if that’s you today as well.

So what did I learn?

I asked myself, could I have done this differently? I’m not really sure. I followed one convention (and if I got that wrong, please let me know in the comments). And what I needed to do was have intimate knowledge of the Golang module system and its limitations. I don’t feel I had the capacity for that at the time. Now I do.

I feel like this is a workflow for someone who is expected not to make mistakes. It promotes a rigidity that I don’t think plays well with code evolution, and certainly not learning.

Sometimes there is too much weight thrown behind semver as well. Who hasn’t happily sat on 0.x.x or 1.x.x for a significant period of time just because its easier than having to explain “a breaking change” ?.

I am also a believer that the refactoring we need to do of our code is never complete. Especially for the work that takes some use to understand – I didn’t have all the answers about what a SPARQL library needed and needed to publish it so others could help contribute to that if they liked.

Golang uses the assumption that packages are “experimental” until a 1.0.0. is released – if that’s the case why semver?

Perhaps I should just stay “experimental” – it sounds cooler.

Either way, it’s a reminder, that no matter how hard you try sometimes you can’t get it right. But also, all “standards” are artificial constructs, it’s a weakness today but you’re going to use that to your strength other days. So, keep trying.

Acknowledgements

• The Golang Gopher at the top of this post was drawn and created by artist Renée French.