{"id":1465,"date":"2023-10-20T23:32:29","date_gmt":"2023-10-20T22:32:29","guid":{"rendered":"https:\/\/exponentialdecay.co.uk\/blog\/?p=1465"},"modified":"2025-12-01T16:59:54","modified_gmt":"2025-12-01T16:59:54","slug":"shattering-the-eyeglass","status":"publish","type":"post","link":"https:\/\/exponentialdecay.co.uk\/blog\/shattering-the-eyeglass\/","title":{"rendered":"Shattering the eyeglass: Using Kaitai Structs to dissect the eyeglass’ contents"},"content":{"rendered":"

In my post from 2012: Genesis of a File Format<\/a>, I created a new file format – the Eyeglass<\/em> file format<\/a>. The format provides a mechanism to persist information about a patient’s eye health following a checkup at an opticians. Today in 2023 we can use the format to understand how to make use of Kaitai Structs<\/a> for understanding file formats.<\/p>\n

Given the disclaimer that I am not actually an optician and that the format is purely illustrative, let’s look at the eyeglass again below.<\/p>\n

<\/p>\n

A recent post on digipres.club<\/a> prompted me to revisit this work, starting with refactoring the original GitHub repository<\/a> and leading to this blog post where I thought it might be an interesting idea to break down the binary format that is output by the tool.<\/p>\n

Thus, in the first blog<\/a> we create the binary format from a specification. In this blog we try and understand the format from the binary itself – more akin to a transfer and digital preservation workflow in an archival institution.<\/p>\n

Breaking down the eyeglass<\/h2>\n

The binary file output in the eyeglass project looks as follows (xxd<\/code> allows me to pretty print the hex (hexadecimal)):<\/p>\n

xxd -g1 prescription-sample-1.0-be.eygl<\/pre>\n
00000000: bb 0d 0a 65 79 65 67 6c 61 73 73 1a 0a ab 01 01 ...eyeglass.....\r\n00000010: 32 30 31 32 2d 31 31 2d 30 38 54 31 32 3a 33 37 2012-11-08T12:37\r\n00000020: 3a 35 30 00 00 00 00 00 00 00 00 00 00 00 00 00 :50.............\r\n00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n00000070: 00 00 00 00 00 00 00 00 00 00 00 c0 56 66 66 3f ............Vff?\r\n00000080: 00 00 00 be 80 00 00 bf 80 00 00 00 00 00 82 00 ................\r\n00000090: 00 00 50 00 00 00 00 00 00 00 00 00 00 00 00 00 ..P.............\r\n000000a0: 00 00 00 3f 28 f5 c3 3f 00 00 00 00 00 00 0c 00 ...?(..?........\r\n000000b0: 00 00 0c 44 69 73 74 61 6e 63 65 20 61 6e 64 20 ...Distance and\r\n000000c0: 43 6c 6f 73 65 20 57 6f 72 6b 2e 00 00 00 00 00 Close Work......\r\n000000d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n000000e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n00000100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n00000110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n00000120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50 ...............P\r\n00000140: 61 74 69 65 6e 74 27 73 20 65 79 65 73 69 67 68 atient's eyesigh\r\n00000150: 74 20 6e 65 65 64 73 20 63 6f 72 72 65 63 74 69 t needs correcti\r\n00000160: 6f 6e 2e 20 48 69 73 74 6f 72 79 20 6f 66 20 64 on. History of d\r\n00000170: 69 61 62 65 74 65 73 20 69 6e 20 66 61 6d 69 6c iabetes in famil\r\n00000180: 79 20 62 75 74 20 69 6e 64 69 63 61 74 6f 72 73 y but indicators\r\n00000190: 20 66 6f 75 6e 64 2e 20 53 74 61 6e 64 61 72 64 found. Standard\r\n000001a0: 20 63 68 65 63 6b 75 70 20 69 6e 74 65 72 76 61 checkup interva\r\n000001b0: 6c 20 72 65 63 6f 6d 6d 65 6e 64 65 64 2e 00 00 l recommended...\r\n000001c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n000001d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n00000200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n00000210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n00000220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................\r\n00000230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 3f 80 ..............?.\r\n00000240: 00 00 bb 65 6f 66 ...eof\r\n<\/pre>\n
The specification was first described as follows:<\/p>\n
\n
Eyeglass Format Specification 1.0\r\n---\r\nMagic number - 14 bytes - String\r\nVersion - 1 bytes - Unsigned Char\r\nBig-endian - 1 byte - Bool\r\nDate\/time - 19 bytes - String #YYYY-MM-DDTHH:MM:SS\r\nExpansion room - 88 bytes - Undefined\r\nSphere - 8 bytes - R: Float L: Float\r\nCylinder - 8 bytes - R: Float L: Float\r\nAxis - 8 bytes - R: Integer L: Integer\r\nPrism - 8 bytes - R: Float L: Float\r\nBase - 8 bytes - R: Float L: Float\r\nDistance acuity - 8 bytes - R: Float L: Float\r\nNear acuity - 8 bytes - R: Integer L: Integer\r\nPurpose - 140 bytes - String\r\nObservation - 255 bytes - String\r\nNext checkup - 4 bytes - Float\r\nEnd of file - 4 bytes - String\r\n<\/pre>\n<\/div>\n
Between the binary object and this specification we have the two lenses through which we see file formats in digital preservation.<\/div>\n
<\/div>\n
Reading the original blog is a good idea for background, but a summary is that a file format is persisted by writing sequences of bytes to disk. The sequences are bounded by the the data type and respective data length that they represent (a boolean is one byte, a float is 4-bytes, and so on — their own specifications are dictated by the software encoding them, e.g. a 16-bit software vs. 32-bit software, golang vs. rust) – in total, the eyeglass format is written to disk taking up 582 bytes. If we draw the boundaries between the different data types described in the specification it looks as follows:<\/div>\n
\"\"<\/a><\/p>\n
The different data types and their lengths are like fields, each field can be interpreted in some way that may be useful to the original rendering application, and of course to the digital preservation researcher trying to gleam information from a file format.<\/p>\n
But what does that meaning look like?<\/p>\n
Manual decoding of the eyeglass<\/h2>\n
Lets take a small number of the fields in the eyeglass format and take a look.<\/p>\n
Date\/time<\/h3>\n
Date\/time is the date and time that the file was written. It’s a plain old string and uses the ISO standard date format. In hexadecimal it looks as follows:<\/p>\n
32 30 31 32 2d 31 31 2d 30 38 54 31 32 3a 33 37 3a 35 30<\/pre>\n
Which renders as: 2012-11-08T12:37:50<\/em><\/p>\n
When we talk about endianness shortly, this field, as a string, like other string fields in this format doesn’t change.<\/p>\n
Big-endian<\/h3>\n
The big-endian field was intended to be a flag that tells the reader of the format if they are looking at a big-endian version, or a little-endian version. This tells the reader how to interpret the values.<\/p>\n
In hex we simply have a one-byte “boolean” reading as:<\/p>\n
01<\/pre>\n
This should evaluate as “1” or “true” and tells us that we do need to interpret this file’s contents as big-endian.<\/p>\n
Sphere<\/h3>\n
Sphere is documented as containing two values, one for the right eye, and one for the left. The value are floating point numbers. In hexadecimal we have:<\/p>\n
c0 56 66 66 3f 00 00 00<\/pre>\n
The two values: “c0 56 66 66” and “3f 00 00 00”<\/p>\n
The easiest way to understand what these values are, knowing that they are floating point numbers is to use an online hexadecimal converter, e.g. online-hex-converter<\/a>.<\/p>\n
We know it is a big-endian number, we can look up the corresponding fields after data entry:<\/p>\n
\"\"<\/a><\/p>\n
The interpretation “Float – Big Endian (ABCD)” provides the two results we expect here: -3.35; and 0.5 respectively.<\/p>\n
You can get a feel for the issues that might arise interpreting these values without knowing the correct endianness by looking at some of the other results in this hexadecimal converter tool.<\/p>\n
Near acuity<\/h3>\n
Near acuity contains two values which in the specification are two integers, each four bytes: “00 00 00 0c” and “00 00 00 0c”. Knowing these are integers makes\u00a0 their interpretation a bit easier, we can type these into a search engine as follows:<\/p>\n
“0x0000000c in decimal” (the “0x” denotes hexadecimal<\/a> or base-16)<\/em><\/p>\n
A result of 12 will be returned. 0x0c is simply 12 in hex. In big-endian this is an easy conversion no matter how many hex values are here.<\/p>\n
In the hexadecimal converter we can confirm our result:<\/p>\n
\"\"<\/a><\/p>\n
Endianness in a file format<\/h2>\n
Endianness<\/a> changes layout of the bytes and thus how they need to be interpreted. There’s not a huge visual difference between the big-endian version of the eyeglass format and the little-endian version, but the differences are big enough to change the meaning we try to extract from the file.<\/p>\n
We can see the fields that have been changed in one rendering of the big-endian file vs. the little-endian one using biodiff<\/a> (click on the image below if you need to see it better).<\/p>\n
\"\"<\/a><\/p>\n
Interpreting the fields in little endian<\/h3>\n
Taking the example fields from before we can see the differences, and how they look.<\/p>\n\n\n\n\n\n\n\n
Field<\/th>\nBig-endian<\/th>\nLittle-endian<\/th>\nValue<\/th>\n<\/tr>\n
Date\/time<\/td>\n32 30 31 32 2d 31 31 2d 30 38 54 31 32 3a 33 37 3a 35 30<\/td>\n32 30 31 32 2d 31 31 2d 30 38 54 31 32 3a 33 37 3a 35 30<\/td>\n2012-11-08T12:37:50;<\/td>\n<\/tr>\n
Big-endian<\/td>\n01<\/td>\n00<\/td>\n1=True; 0=False;<\/td>\n<\/tr>\n
Sphere<\/td>\nc0 56 66 66; 3f 00 00 00<\/td>\n66 66 56 c0; 00 00 00 3f<\/td>\nR:-3.35; L:0.50;<\/td>\n<\/tr>\n
Near acuity<\/td>\n00 00 00 0c; 00 00 00 0c<\/td>\n0c 00 00 00; 0c 00 00 00<\/td>\nR:12; L:12;<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Let us take the value for near acuity here: “0c 00 00 00”. It is reversed compared to the big-endian version – but if we ask a search engine for “0x0c000000 in decimal” we will return “201326592” which is definitely not the right value! We need to know that little-endian puts the least significant byte at the first memory address (which equates to an earlier position in a file), and in hexadecimal, like decimal, least significant values are the 1s and the 10s. To interpret this value correctly we need to reverse the bytes, so 0x0000000c like we had in big-endian.<\/p>\n
Kaitai<\/h2>\n
Kaitai is an example of tooling we can adopt in our work to automatically parse the structures of a binary format.<\/p>\n
\"\"<\/a><\/p>\n
The animated graphic here shows how kaitai parses the eyeglass format and recognizes the data type boundaries (fields) we’ve been discussing in the examples above.<\/p>\n
Kaitai<\/a> is described as a domain specific language concerned with the parsing of arbitrary binary formats.<\/p>\n
Further, in their introductory text:<\/p>\n
Parsing binary formats is hard, and there\u2019s a reason for that: such formats were designed to be machine-readable, not human-readable. Even when one\u2019s working with a clean, well-documented format, there are multiple pitfalls that await the developer: endianness issues, in-memory structure alignment, variable size structures, conditional fields, repetitions, fields that depend on other fields previously read, etc, etc, to name a few.<\/p>\n
Kaitai Struct tries to isolate the developer from all these details and allow them to focus on the things that matter: the data structure itself, not particular ways to read or write it.<\/p><\/blockquote>\n
Parsing eyeglass in Kaitai<\/h3>\n
We know how the eyeglass file format is structured. We helpfully (!) have a list of the data-types and sizes and what those mean. We can translate those in to the “ksy” YAML<\/a> “kaitai struct language<\/a>” format consumed by kaitai to allow kaitai conformant tools to parse the binary format and to identify the field boundaries and attach meaning to those.<\/p>\n
Given the right instructions kaitai is clever enough to understand how to interpret endianness and the values within certain fields. The format incorporates metadata in the form of a header, and additional documentation strings that can also be used by kaitai tools to output more information to the user.<\/p>\n
A small snippet from the kaitai structs created for the eyeglass format:<\/p>\n
meta:\r\n  id: eyeglass_format\r\n  file-extension: eygl\r\n  xref:\r\n    wikidata: Q105858419\r\n  endian: be\r\n  tags:\r\n    - digipres\r\n  license: CC-BY-SA\r\ndoc: |\r\n  The eyeglass format...\r\nseq:\r\n  - id: magic\r\n    size: 14\r\n    type: str\r\n    encoding: utf8\r\n  - id: version\r\n    size: 1\r\n  - id: endianness\r\n    type: s1\r\n  - id: datetime\r\n    size: 19\r\n    type: str\r\n    doc: \"the date this file was created\"\r\n    encoding: utf8<\/pre>\n
Field types can be defined as well as field lengths.<\/p>\n
In the snippet above we have strings (str); signed 1-byte integers (s1); and encodings (utf-8). If a field wasn’t known but a field length could be determined then size would be enough to start to parse the data and kaitai’s tooling can start to guess at the values to help with reverse engineering efforts.<\/p>\n
We can parse the format with kaitai using the tool kaitai-struct-visualizer<\/a>.<\/p>\n
and the command:<\/p>\n