{"id":268,"date":"2012-11-14T01:24:17","date_gmt":"2012-11-14T01:24:17","guid":{"rendered":"http:\/\/exponentialdecay.co.uk\/blog\/?p=268"},"modified":"2025-12-01T16:59:56","modified_gmt":"2025-12-01T16:59:56","slug":"hacking-the-droid-signature-file-for-characterization","status":"publish","type":"post","link":"https:\/\/exponentialdecay.co.uk\/blog\/hacking-the-droid-signature-file-for-characterization\/","title":{"rendered":"Hacking the DROID Signature File for Characterization"},"content":{"rendered":"

Identification of a format can be approached from many angles. Often a magic<\/a>
\n number<\/a> will be used at the beginning of a file. This may be strengthened by\u00a0the addition of similarly consistent bytes at the end of a file or indeed any\u00a0part of the bitstream inbetween. Using the sample file format we created\u00a0last week<\/a>\u00a0the magic number to identify it is specified as:<\/p>\n

'\\xBB\\x0D\\x0A\\x65\\x79\\x65\\x67\\x6C\\x61\\x73\\x73\\x1A\\x0A\\xAB'<\/pre>\n
Alone this translates to a DROID<\/a> signature as follows:<\/p>\n
Position: Absolute beginning of file (BOF)\r\nSequence: BB0D0A657965676C6173731A0AAB\r\nMax Offset: 0<\/pre>\n
To strengthen such a signature we may also make use of the end of file<\/em>\u00a0(EOF) sequence and as the file is of fixed size (582 bytes) we could also add an {n}-bytes<\/em> wildcard parameter between the BOF and EOF sequences to ensure DROID only matches files of a particular size which contain both signatures. My usual approach is to create two separate sequences to ensure the EOF is found as expected at the end of the bit stream. Given the specification of the Eyeglass format contains a version field I will also include that in the BOF sequence:<\/p>\n
Eyeglass Version 1.0: Signature\r\n---\r\nPosition: Absolute beginning of file (BOF)\r\nSequence: BB0D0A657965676C6173731A0AAB01\r\nMax Offset: 0\r\n\r\nPosition: Absolute end of file (EOF)\r\nSequence: BB656F66\r\nMax Offset: 0<\/pre>\n
Identification aside, signatures can encapsulate a lot of information. The detail of which we lose but which could potentially be useful. Manipulating DROID signatures allows us to use DROID to\u00a0return this more detailed\u00a0characterization<\/em>\u00a0information. A common scenario might be\u00a0the spotting files without their terminating end of file sequences, PDF and\u00a0JPEG2000, two examples where EOF sequences feature prominently.<\/p>\n
If a format specifies that it must<\/em> begin with an BOF sequence and must<\/em> terminate\u00a0with an EOF sequence to be considered ‘valid’ there is a preservation risk\u00a0in a tool making an assertion about the identification of an ‘invalid’\u00a0format where either of these sequences do not appear. With validation existing in a preservation workflow to mitigate this risk, identification tools really just provide clues as to what digital objects we are looking at; even though\u00a0strict\u00a0<\/strong><\/em>identification of formats might be preferred (a commentary on which is beyond the scope of this blog entry).<\/p>\n
An alternative to simply encapsulating combinations of sequence, with, or without end of file markers into a single signature is to use the <HasPriorityOverFileFormatID><\/strong> element\u00a0<\/strong>in the DROID signature file to create two<\/strong> signatures, one for each scenario. Priorities enable us to return more detailed characterization information about our results as appropriate:<\/p>\n
<FileFormat ID=\"1\" Name=\"Eyeglass Format: BOF and EOF\" PUID=\"id\/1\">\r\n <InternalSignatureID>1<\/InternalSignatureID>\r\n <Extension>eygl<\/Extension>\r\n <HasPriorityOverFileFormatID>2<\/HasPriorityOverFileFormatID><\/strong><\/span><\/span>\r\n<\/FileFormat>\r\n<FileFormat ID=\"2\" Name=\"Eyeglass Format: No EOF\" PUID=\"id\/2\">\r\n <InternalSignatureID>2<\/InternalSignatureID>\r\n <Extension>eygl<\/Extension>\r\n<\/FileFormat><\/pre>\n
DROID will correctly identify each file but it won’t output multiple identifications:<\/p>\n
\"\"<\/a>
DROID run identifying Eyeglass versions with both BOF and EOF sequences and without its EOF sequence.<\/figcaption><\/figure>\n
We can delve deeper into characterization using DROID and start creating signatures for other features we might want information about and might cause issues interpreting the file in future. Given the eyeglass format specification talks about endianness we can imagine three scenarios for which characterization signatures<\/em> could be created:<\/p>\n