Shortened links? Expand them and save the URLs

Shortened links are a digital preservation and web archiving nightmare. You can imagine how they need to work:

1. Create a unique short code for a given (target) URL (like a hash, but far far shorter)
2. Pair the short code with your URL in a database.
3. Create a redirect rule on the URL shortening server from the new source URL to the target URL.
4. Send the shortened link to the caller, e.g. shortURL.com/123badf00d
5. In-perpetuity: continue to pay for your domain; maintain the database; look after redirect rules during server migrations; ensure duplicate short-codes are not created.

A URL-shortening business in five easy steps.

But what does that mean for digital preservation?

URL shortening and digital preservation

There are different degrees of issue with URL shortening.

1. The reality is that databases of redirects are not maintained and links disappear.
2. Shortened links obfuscate meaning and may be harmful.

The problem of (2) can happen with regular hyperlinks, especially with domain squatting, but domain squatting aside, there’s usually some authority, and thus provenance, associated with a domain name and its top-level-domain, e.g. you should usually be fine with .gov .edu and so on.

NB. I don’t have numbers on skew. Skew is usually the phenomenon of content changing behind a hyperlink after it was first published, but with short URLs, skew might mean pointing to two different domains over time. I have only seen something similar with QR codes that used an additional proxy in between the redirect to enable the target URL to be updated and replaced multiple times; but this is not a dissimilar mechanism from a URL shortening service.

Can I get more information?

Take a bit.ly URL that works today… bit.ly/2lfnTqv

Use cURL to get the header information with the -I flag: curl -I bit.ly/2lfnTqv

HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Thu, 02 Jul 2026 17:47:26 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 130
Cache-Control: private, max-age=90
Content-Security-Policy: referrer always;
Location: http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/
Referrer-Policy: unsafe-url
Set-Cookie: _bit=q62hLq-038ddea2565856aace-00S; Domain=bit.ly; Expires=Tue, 29 Dec 2026 17:47:26 GMT
X-Robots-Tag: noindex
Via: 1.1 google

I have highlighted three key pieces of information that we really need to pay attention to:

• HTTP redirect code and reason: HTTP/1.1 301 Moved Permanently
• Location (target) of the redirect: Location: http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/
• And the source URL, the shortened link: bit.ly/2lfnTqv

The redirect here is 301: Moved Permanently, This belongs to a class of “3.x.x” HTTP response codes and they all deal with content moving in some way. You can read more about 301 on the Mozilla Website. The Mozilla site also lists the other 3.x.x codes, which you might also see in the wild, given that their implementation is usually up to a developer’s interpretation and, thus, between them and however their coffee is helping them see the world today.

The location (target URL) is, of course, the URL that we were always interested in. It is essentially whatever digital cultural artifact we were reading was pointing to.

The source URL is called so (in this blog) because it is what we received and what we extracted from the aforementioned digital cultural artifact.

To ensure we capture everything, we need:

• A record of the source URL.
• A record of the target URL and its connection to the source URL.
• A web archive of the target URL.

This helps protect against content disappearing or becoming skewed. It also ensures that if we ever see the short URL in other records, we know what it pointed to, in case one day we can no longer rely on the redirect from the original link shortening domain.

In an ideal world! It might also, one day, help other archives cross-reference their short URLs, but I don’t think many of our archives and preservation systems are set up to enable this… but I like to dream.

Capture in the Internet Archive

The Internet Archive (IA) has a small flaw here.

Capture is all in the user-interface

As you can see by the screenshot below you can capture a shortened link, but there’s no easy way to preserve this information in a digital preservation system and preserve a functional audit trail.

Capture using cURL and JC

You want to capture more information about the original URL — in this instance both additional metadata and paradata (data about the technical processes contributing to a record). While you still need to create an archiving step, you can extract machine-readable information from cURL by combining it with the jc tool which converts informatiion on the terminal to JSON; you can read more on GitHub.

Let’s look at an example: jc curl -Is bit.ly/2lfnTqv | jq

[
  {
    "_type": "response",
    "_response_version": "HTTP/1.1",
    "_response_status": 301,
    "_response_reason": [
      "Moved Permanently"
    ],
    "server": [
      "nginx"
    ],
    "date": "Thu, 02 Jul 2026 18:14:54 GMT",
    "content-type": "text/html; charset=utf-8",
    "content-length": 130,
    "cache-control": [
      "private",
      "max-age=90"
    ],
    "content-security-policy": [
      "referrer always;"
    ],
    "location": "http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/",
    "referrer-policy": "unsafe-url",
    "set-cookie": [
      "_bit=q62ieS-627d247d0d017ad442-00V; Domain=bit.ly; Expires=Tue, 29 Dec 2026 18:14:54 GMT"
    ],
    "x-robots-tag": "noindex",
    "via": [
      "1.1 google"
    ],
    "date_epoch_utc": 1783016094
  }
]

All of the same information is there from our original call to cURL, but via JC it is now machine readable.We can copy this data verbatim into a provenance record or use another tool to cherry-pick the information we need into a record about the web archiving process.

Take this call to cURL, tag on a request to your local web-archiving provider for a memento snapshot, and you’re good! 8-wire web-archiving in action!

But is there another way?

HTTPreserve

I have discussed components of HTTPreserve before. HTTPreserve is a service and suite of tools I wrote a while back to provide a standard web-archiving export, especially for when links were found in the records already stored in the digital archive and needed to be preserved.

HTTPreserve can check for the existence of an already recorded IA snapshot, but it also has a save function that saves the most up-to-date snapshot as well as returns machine-readable information describing the link’s status.

E.g. call:

curl -s https://httpreserve.info/save?url=https://bit.ly/2lfnTqv

or access the URL in the browser and you will see something along the lines of:

{
   "AnalysisVersionNumber": "0.0.20",
   "AnalysisVersionText": "exponentialdecay-httpreserve/0.0.20",
   "SimpleRequestVersion": "httpreserve-simplerequest/0.0.4",
   "Link": "http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/",
   "Title": "",
   "ContentType": "",
   "ResponseCode": 301,
   "ResponseText": "Moved Permanently",
   "SourceURL": "http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/",
   "ScreenShot": "snapshots are not currently enabled",
   "InternetArchiveLinkEarliest": "https://web.archive.org/web/20170908143755/http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/",
   "InternetArchiveEarliestDate": "2017-09-08 14:37:55 +0000 UTC",
   "InternetArchiveLinkLatest": "https://web.archive.org/web/20260703163229/http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/",
   "InternetArchiveLatestDate": "2026-07-03 16:32:29 +0000 UTC",
   "InternetArchiveSaveLink": "https://web.archive.org/save/http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/",
   "InternetArchiveResponseCode": 302,
   "InternetArchiveResponseText": "Found",
   "RobustLinkEarliest": "<a href='https://web.archive.org/web/20170908143755/http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/' data-originalurl='http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/' data-versiondate='2017-09-08'>HTTPreserve Robust Link - simply replace this text!!</a>",
   "RobustLinkLatest": "<a href='https://web.archive.org/web/20260703163229/http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/' data-originalurl='http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/' data-versiondate='2026-07-03'>HTTPreserve Robust Link - simply replace this text!!</a>",
   "PWID": "urn:pwid:archive.org:2026-07-03T16:32:29Z:page:http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/",
   "Archived": true,
   "Error": false,
   "ErrorMessage": "",
   "StatsCreationTime": "1.975691615s"
}

The information returned here includes:

• Link which is where the short URL pointed at, our target URL.
• Source URL (SourceURL), i.e. the shortened URL.
• Web archive latest link (InternetArchiveLinkLatest) and earliest link (InternetArchiveLinkEarliest).
• Robust Link (RobustLinkLatest and RobustLinkEarliest)snippets for use in your own HTML.
• Persistent Web Identifier (PWID) (spec)

HTTPreserve also constructs an IA save link (InternetArchiveSaveLink) that you can use yourself:

https://web.archive.org/save/http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/

All of this information could be used to automate archiving workflows and commit information directly to your digital preservation audit trails with additional provenance.

NB. Saving via HTTPreserve can time out, and you might just want to see if a short link has been preserved before. Just replace save with httpreserve:

curl -s https://httpreserve.info/httpreserve?url=https://bit.ly/2lfnTqv

Linkstat

I wrote linkstat to bring httpreserve to the command line. You can download linkstat and run the following command to save a web resource:

./linkstat -link https://bit.ly/2lfnTqv -save

Or just to view:

./linkstat -link https://bit.ly/2lfnTqv

And the result should look exactly as before:

{
   "AnalysisVersionNumber": "0.0.21",
   "AnalysisVersionText": "exponentialdecay-httpreserve/0.0.21",
   "SimpleRequestVersion": "httpreserve-simplerequest/0.0.4",
   "Link": "http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/",
   "Title": "",
   "ContentType": "",
   "ResponseCode": 301,
   "ResponseText": "Moved Permanently",
   "SourceURL": "https://bit.ly/2lfnTqv",
   "InternetArchiveLinkEarliest": "https://web.archive.org/web/20170908143755/http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/",
   "InternetArchiveEarliestDate": "2017-09-08 14:37:55 +0000 UTC",
   "InternetArchiveLinkLatest": "https://web.archive.org/web/20260703163229/http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/",
   "InternetArchiveLatestDate": "2026-07-03 16:32:29 +0000 UTC",
   "InternetArchiveSaveLink": "https://web.archive.org/save/http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/",
   "InternetArchiveResponseCode": 302,
   "InternetArchiveResponseText": "Found",
   "RobustLinkEarliest": "<a href="https://web.archive.org/web/20170908143755/http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/" data-originalurl="http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/" data-versiondate="2017-09-08">HTTPreserve Robust Link - simply replace this text!!",
   "RobustLinkLatest": "<a href="https://web.archive.org/web/20260703163229/http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/" data-originalurl="http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/" data-versiondate="2026-07-03">HTTPreserve Robust Link - simply replace this text!!",
   "PWID": "urn:pwid:archive.org:2026-07-03T16:32:29Z:page:http://choosestainless.outokumpu.com/case-barbarastollen-underground-archive/",
   "Archived": true,
   "Error": false,
   "ErrorMessage": "",
   "StatsCreationTime": "2.89741131s"
}

NB. Saving can take time, I’d recommend first using linkstat -link <your link> to see if archived copies already exist, especially for URLs in older collections. Most of the latency is on the IA’s side because it needs to rate-limit requests, but there might be a way around this with a bit more investment in the linkstat code. We can talk if this is of interest to you.

Deliberate at all times

Give HTTPreserve and linkstat a whirl and let me know how it goes. Even if you don’t follow up with those, I hope some of the tips in this blog will prove helpful, and you’re already thinking about how to store more information about your web archives and your digital preservation processes.

What other gaps are you already thinking about in your archives?

Further reading

A good URL shortener?

I still have it on my backlog to come back to this article, “Building a Less Terrible URL Shortener” and maybe trying to implement some of its ideas.

• https://blog.legoktm.com/2024/07/23/building-a-less-terrible-url-shortener.html

Binary Trees

HTTPreserve came out of my work on my Binary Trees paper and describes a number of pragmatic approaches to automatically connecting born digital archives. This blog is yet another example of how we need to connect the information we have, source URL to target URL and persist this information when it is available.

• Binary Trees? Automatically Identifying the Links Between Born-digital Records

HTTPreserve suite

The HTTPreserve suite of tools is available on GitHub.