I often describe what I do as GLAM data plumbing. Most of the time I’m not creating new tools, I’m figuring out what data is available and how I can connect it up to existing tools. It’s rarely straightforward, but if I can get all the pipes connected and data flowing in the right direction, suddenly new things become possible.

Things like turning all the State Library of Victoria’s digitised maps into data.

I’ve just created a workflow that uses AllMaps and IIIF to georeference the SLV’s digitised maps. There’s some technical details below, but the idea is pretty simple. A userscript links the SLV image viewer to Allmaps – so you just click on a button, and the digitised map opens, ready for georeferencing.

Why is this useful? Georeferencing relates a digitised map to real world geography. It describes the map’s position and extent using geospatial coordinates – turning historic documents into geospatial data that can be indexed, visualised and manipulated. Georeferencing opens digitised maps to new research uses.

So, how many maps we can georeference before my residency finishes in December? Hundreds? Thousands? If you like maps and want to help, head to the documentation page to find out how to get started. And if you want to see how things are progressing, have a look at the project dashboard.

View the documentation to get started

A few technical details follow…

Early on in my time as Creative Technologist-in-Residence at the State Library of Victoria, I started playing around with Allmaps for georeferencing digitised maps. It’s a great tool (really a suite of tools and standards) because instead of constructing a whole new platform it integrates with existing IIIF services. The SLV provides digitised images through IIIF, so I thought it should be possible to use Allmaps to georeference the SLV’s map collection.

But I struck a problem that took some time to unravel. The IIIF urls in the SLV manifests include port numbers and that confused Allmaps. The manifests also sometimes contained references to image formats that weren’t actually accessible, generating errors when they were loaded. Hopefully these problems will be fixed by the SLV, but in the meantime I’ve created a proxy service that edits the manifest on the fly. The proxied urls can be loaded into the Allmaps Editor without errors. Pipes fixed, data flowing!

Using the manifest proxy

To generate a link to a proxied manifest, first grab the item’s IE identifier from the url of the digitised item viewer. For example, the identifier in this url https://viewer.slv.vic.gov.au/?entity=IE15485265&mode=browse is IE15485265. Once you have the identifier, add it to the end of the url https://wraggelabs.com/slv_iiif/. For example, https://wraggelabs.com/slv_iiif/IE15485265. You can then supply this url to the Allmaps editor.

But having to fiddle around with proxies didn’t make a great user experience. I needed some way of integrating the two services, so that a user could just click a button in the SLV website and start editing in Allmaps. Userscripts to the rescue!

I wrote recently about hacking GLAM collection interfaces using userscripts. Since I started my residency at the SLV, I’ve also created a userscript to display the IIIF manifest url in the SLV image viewer, and run a Code Club workshop where we played around with an assortment of SLV website hacks.

As in a number of these examples, the georeferencing userscript adds new features to the SLV website, but there’s a fair bit more going on under the hood. It runs automatically every time you load the SLV image viewer, and then:

• it checks the metadata of the digitised item to see it it’s a map (or something that contains maps, like an atlas or street directory)
• if it looks like a map, it generates an Allmaps identifier using the item’s IIIF manifest url and checks with Allmaps to see whether the item has already been georeferenced
• it adds a ‘Georeferencing’ section to the page, with a button to georeference the item (or edit the existing georeferencing)
• if the item has already been georeferenced, it adds a button to view the item in the Allmaps Viewer, and embeds a live preview Accessing metadata

The userscript gets the item metadata from a JSON file that’s loaded by the image viewer. The JSON file includes a lot of extra, useful information about the digitised item. To access the JSON file, you just construct a url like this: https://viewerapi.slv.vic.gov.au/?entity=[IE identifier]&dc_arrays=1. The IE identifier is in the url of the image viewer.

Allmaps identifiers

Allmaps creates its identifiers by hash encoding the IIIF urls. The userscript borrows some code from the Allmaps id module to generate the ids, then sends a HEAD request to the Allmaps API to see whether an entry for the current manifest exists.

Example of an item that hasn’t been georeferenced yet Example of an item that has been georeferenced, displaying an embedded version of the Allmaps viewer

I’ve also created a GitHub repository to save copies of the data. Every two hours this notebook is run to query the Allmaps API for newly georeferenced maps. These are added to a dataset which is saved in three formats:

• a CSV file
• a CSV file that includes thumbnails and links for viewing in Datasette-Lite
• a GeoJSON file, that can be viewed in services like geojson.io

At the same time, the data for each individual map is downloaded and saved as IIIF annotations (in JSON) and GeoJSON.

Finally, this notebook is run to generate a dashboard that provides an overview of the project’s progress.

The project dashboard is updated every two hours

One of the Allmaps developers described all my plumbing and workarounds as a ‘very cool lofi example of how you can set this up with little means’, and I think that’s pretty apt. It’s really just an experiment to demonstrate the possibilities, but by connecting up existing services it’s generating real data of long term value.