{"id":1714,"date":"2024-07-15T07:12:19","date_gmt":"2024-07-15T07:12:19","guid":{"rendered":"https:\/\/exponentialdecay.co.uk\/blog\/?p=1714"},"modified":"2024-10-16T10:01:53","modified_gmt":"2024-10-16T10:01:53","slug":"we-love-our-fiefdoms-the-dpc-and-wikidata","status":"publish","type":"post","link":"https:\/\/exponentialdecay.co.uk\/blog\/we-love-our-fiefdoms-the-dpc-and-wikidata\/","title":{"rendered":"We love our fiefdoms: The DPC and Wikidata"},"content":{"rendered":"

Andy Jackson is doing some positive work for the DPC, especially in his low-level look at Wikidata recently<\/a> (external blog).<\/p>\n

Specifically, his work on a SPARQL query<\/a> we use in Siegfried might just solve a problem we have with downloading Wikidata results<\/a> in the United States (and maybe elsewhere).<\/p>\n

In his blog post (above) he describes some issues with the results returned from the existing file formats SPARQL query, specifically gaps in the results. Issues having full knowledge of some of the syntax used in the query language (very relatable); and ends with a request for help, which we will get to below.<\/p>\n

But first, there are two red flags I’ve noticed in this work:<\/p>\n

    \n
  1. When the Wikidata File Format Query snippet was first crimped from Siegfried without acknowledgement or engagement: https:\/\/github.com\/digipres\/sentinel\/issues\/13#issuecomment-1235451598<\/a><\/li>\n
  2. The suggestion in Andy’s post above someone is ‘doing it wrong’ (I read the someone as the Wikidata community).<\/li>\n<\/ol>\n

    On 1. as a developer there are things that take minutes to work out, and things that take days (Andy’s own getting to grips with Wikidata probably reflects some of my own learning efforts). The query for Siegfried took days to develop and overall weeks to test, refine, iterate, and make sensible for use in the Siegfried Harvest process. While it is only an example of declarative programming<\/a>, it’s license unclear in the scheme of things, a more overt shout-out to the project and more open communication about aligning efforts would be nice – and given the role of the Regstries of Good Practice Project<\/a>, more visibility about extensive<\/em> existing efforts.<\/p>\n

    Wikidata: A Magic Portal for Siegfried and Roy<\/h2>\n

    On 2. I have shared my views about owning the file format problem previously<\/a>.<\/p>\n

    I have also been part of a more specific call to the community in the iPRES paper Wikidata: A Magic Portal for Siegfried and Roy<\/a>.<\/p>\n

    Specifically under “Creating a together community”<\/em>:<\/p>\n

    The corollary to engagement with the integration is that the digital preservation community also needs to learn how to engage with Wikidata in ways that are sympathetic with the Wikidata community guides and guidelines. We are potentially two communities that need to come together, not a digital preservation community squatting in a Wikidata world.<\/p>\n

    Wikidata makes it very easy to edit entries. It offers a discussion platform to discuss those changes too. Where new properties are added to the Wikidata schema then those can be discussed in forums of in creasing importance as well. While the service is community governed, that governance is somewhat different to the idea of a central point of focus to the registries of the past.<\/p>\n

    There is a certain amount of \u201cWikidata literacy\u201d that needs to be developed alongside this resource to help make it work. In this sense two communities can become a \u201ctogether\u201d community with that of \u201cdigital preservation\u201d branching out beyond our national memory institutions or vendors as leaders in our community registry work, to that of the wider technology \ufb01eld. An unintended consequence of that effort too is that the wider technology world may have resources to help bolster \u201cour\u201d efforts.<\/p>\n

    People who would like to contribute \ufb01le format sig-nature information to Wikidata are welcome to add it. Digital Preservationists could share signature data via Wikidata, and could even use the platform to discuss with one another. Each item in Wikidata has an associated \u201cDiscussion” page” just as each article in Wikipedia does. If someone needs clari\ufb01cation about how a signature was determined it is possible to ask that question on Wikidata and see if others reply.<\/p><\/blockquote>\n

    In earlier drafts and maybe my iPRES presentation on this, it was very much framed as we cannot simply colonize<\/em> Wikidata. We must recognize that there are many smart eyes looking at this shared service, and who may indeed think ‘we’<\/em> are doing it wrong.<\/p>\n

    Avoiding our colonization of the service does not begin with us saying someone is doing it wrong, we need to recognize it’s a conversation, and a process.<\/p>\n

    Wikidata anxiety<\/h2>\n

    With some of my own anxiety as a professional (trying to have my own work visible to institutions and a community bigger than myself) exposed above; I recognize in Andy’s request for help, an anxiety – specifically in how to engage with the Wikidata community (Wikidatans).<\/p>\n

    I have huge amounts of sympathy for this. It’s a different world from digital preservation. There are thousands of people engaged with this service daily, and of course, they’re not all engaged with file formats like we are.<\/p>\n

    My first reference for Andy is to look at the work of one of my colleague’s on the Siegfried Wikidata project Dr. Thornton in Wikidata for Digital Preservationists<\/a> a technical guide they wrote for the Digital Preservation Coalition (DPC). The guide should provide some sense of confidence about engaging with Wikidatans.<\/p>\n

    My second is to refer back to the similar questions myself and colleagues raised in Wikidata: A Magic Portal for Siegfried and Roy<\/a>; and some of the tips I have around discussions in Wikidata and getting started.<\/p>\n

    My third, is to refer Andy to a repository I created for some of these issues, WikiDP-Issues<\/a>. I created this repository back in 2020 so that digital preservationists could have a space to talk about Wikidata safely outside of the Wikidata community and then consider the best approach to take matters to Wikidata — or the other way, to our own registry maintainers and so on. I hoped to compartmentalize the anxiety about the breadth of the Wikidata dataset and its participants into manageable chunks and tackle them piece by piece — I also wanted to create this space to ask others for help, but my bigger hope was that the community would see GitHub as a community space, for building a community together, in this case, tackling the Digital Preservation Wikidata issues together.<\/p>\n

    Linting and the quality of Wikidata data<\/h3>\n

    What metric is most important when we look at file formats in Wikidata?<\/em><\/p>\n

    With the changes made in the most recent pull-request<\/a> to Siegfried.<\/p>\n

    Linting<\/a> .\/roy build -wikidatadebug<\/code> still shows 195 issues outstanding in the data Siegfried tries to consume.<\/p>\n

    {<\/code>
    \n\u00a0 \u00a0\"AllSparqlResults\": 9284,<\/code>
    \n\u00a0 \u00a0\"CondensedSparqlResults\": 8118,<\/code>
    \n\u00a0 \u00a0\"SparqlRowsWithSigs\": 9284,<\/code>
    \n\u00a0 \u00a0\"RecordsWithPotentialSignatures\": 8118,<\/code>
    \n\u00a0 \u00a0\"FormatsWithBadHeuristics\": 46,<\/code>
    \n\u00a0 \u00a0\"RecordsWithSignatures\": 8072,<\/code>
    \n\u00a0 \u00a0\"MultipleSequences\": 12,<\/code>
    \n\u00a0 \u00a0\"AllLintingMessages\": [<\/code>
    \n\u00a0 \u00a0 \u00a0 \"Use the `-wikidataDebug` flag to build the identifier to see linting messages\"<\/code>
    \n\u00a0 \u00a0],<\/code>
    \n\u00a0 \u00a0\"AllLintingMessageCount\": 195,<\/code>
    \n\u00a0 \u00a0\"RecordCountWithLintingMessages\": 156<\/code>
    \n}<\/code><\/p>\n

    These all need to be addressed some way to improve what we’re doing with Siegfried and Wikidata. Linting is used as a term in the programming sense<\/a> and just provides a way to statically analyse the data we want to use. Linting is part of the process of rejecting data that is unsuitable or might cause other harm to Siegfried users, e.g. output completely incorrect format identification results.<\/p>\n

    Andy’s blog, describes one specific issue out of many that need addressing. Some more serious<\/a> than others (from a quality assurance perspective).<\/p>\n

    We need both a (relatively) complete output of the records that Wikidata hosts, as well as<\/em> an understanding of the quality of the data in those records — the former is rendered less useful without the latter.<\/p>\n

    Andy’s request for help<\/h2>\n

    Andy ends his blog with the following questions:<\/p>\n