Using IIIF (with AWS) at SFO Museum

At the end of that first blog post about go-iiif we wrote:

An ideal scenario is one where a museum could upload a set of full-sized images to a AWS S3 bucket, wait for Amazon’s computers to process each image … and then find a new set of images to download (along with a reasonable bill for services rendered) in a different S3 bucket.

Today, that is possible.

This is also where things start to get technical so if you’re not interested in how the proverbial sausage is made you might want to stop here and simply enjoy the punchline which is: More, different images; faster and cheaper.

A quick refresher: IIIF is an acronym for the International Image Interoperability Framework, a project driven by public institutions and private companies in the cultural heritage sector to produce common standards and interfaces (APIs) for accessing and working with collections material.

go-iiif is software written in the Go programming language that implements the IIIF Image API and that SFO Museum has been using to process the images in its collection. Today we are happy to announce the following contributions to the go-iiif project:

• A simple tool called iiif-process and a declarative syntax for creating multiple derivative images in parallel.

This is a thin layer on top of go-iiif to make generating a common set of derivatives (thumbnails, cropped images, and so on) for a large body of source images just a little bit easier. Everything else that follows is the scaffolding required to run these tools on other people’s infrastructure.

• A Dockerfile to run the tool inside of a container.

If you’re not sure what “Dockerfile” or “container” mean it’s easiest to think of the latter as a virtual computer and the former as the instructions (used by an application called Docker) to create it.

• A second tool called iiif-process-ecs to run the iiif-process tool (inside a container) as either an AWS Elastic Compute (ECS) task

• The iiif-process-ecs tool can also be run as an AWS Lambda function to invoke an ECS task (that runs the container (that runs the iiif-process tool)).

If you’re not sure what “Lambda function” means think of it as an atomic piece of code that can be invoked on-demand; sort of like a “baby-container” with fewer capabilities.

That’s a lot to digest so here’s a handy diagram to show how everything fits together and where the layers of separation are defined. Simple, right?

• The yellow boxes represent software in the actual go-iiif package, as well as its dependencies.
• The blue boxes are Docker-related “wrappers”; all the things that handle creating a virtual computer that can run the go-iiif software.
• The pink boxes are the AWS-related wrappers (around the Docker wrappers) as well the different “entry points” to run the go-iiif software inside Amazon’s data centers.

I sometimes joke that Docker is a “flannel shirt” on top of go-iiif and AWS is a “big fuzzy sweater” on top of Docker. There are two reasons for structuring things this way:

First, an ECS task is effectively a shrink-wrapped, on-demand computer (whose specifics are defined by a Dockerfile) that can be spun up to do some work (like processing images) and then torn-down as soon as the work is done.

You can run multiple ECS tasks in parallel, independent of one another, and you are only billed for the time they are running. This is a lot cheaper than running one or more always-on computers inside AWS and a lot easier than having to manage a large workload across those machines yourself. Since every image handled by iiif-process is producing multiple derivatives and any one computer only has a finite number of processors there is a lot to be said for letting Amazon schedule and allocate multiple tasks in the available margins of all the servers and processors they’re already operating.

Second, AWS compliments go-iiif and makes a lot of things easier and cheaper but AWS does not become a requirement in order to use go-iiif.

You can migrate your Docker-related scaffolding to another cloud computing service or simply run things on your own server infrastructure should you choose to do so.

This part is really, really important. Docker-based containers have become a sort of lingua franca for defining and packaging virtual computers. It’s a standard common to all the “cloud services” companies so that helps prevent vendor lock-in. Although I keep saying “Amazon” and “AWS”, and while some things like ECS tasks or Lambda functions are AWS specific, it should still be possible to use the actual iiif-process tool with a Google Cloud or Microsoft Azure or equivalent.

Even if Docker or Dockerfiles go “sour” they are ultimately just one way to package containers (virtual computers) among many. At the end of the day if a packaging standard can make a container that looks and feels and runs like a plain-vanilla Unix computer then the ability to use the go-iiif toolchain is preserved.

It is easy to get distracted by the allure of very real budget savings and reduced infrastructure overhead that cloud-or-serverless-or-just-not-your-own-machines style computing offers. It is equally important, especially for cultural heritage institutions, to have some idea of how things will keep working if a vendor becomes unreliable or worse predatory.

A larger conversation

What all of this points to is a much larger and on-going conversation about the pros and cons, and risks, of outsourcing general purpose computing to services like Amazon. There are a lot of opinions and just as many unique scenarios that dictate the choices people make. Somewhere between “All in!” and “It’s just a tool of The Man designed to keep you down” lies the reality.

Recently I’ve been pointing people to Tim Bray’s Serverlessness blog posts. Tim works for AWS and these blog posts are the by-product of an hour-long talk he did last year explaining the benefits of letting Amazon run the servers (computers) that power your services. Tim’s arguments are not disinterested but he makes good points in AWS’ favour and is refreshingly frank about the remaining stress-points.

A valuable counterpoint to Tim is David Rosenthal’s Cloud For Preservation. This is a very long blog post and focuses mostly on data storage but is absolutely worth reading and the general concerns he raises can be applied to most aspects of “cloud” computing.

If you have the time, go read those blog posts and watch the videos now before life gets in the way. I’ll wait.

Processing images from the command line

So, how does all of this work?

iiif-process is a command-line tool so you invoke it from your computer’s “terminal” application passing it the path to a (IIIF) configuration file, an “instructions” file and one or more images to process.

For example:

$> ./bin/iiif-process -config config.json -instructions instructions.json -uri source/IMG_0084.JPG

Reminder: Both the root of your source images and the location where derivatives are created are both defined in the IIIF config file.

“Instructions” files are JSON-encoded dictionaries of user-defined labels and one or more IIIF Image API image request parameters. There is no limit (beyond what your servers can handle) to the number of instructions you can define in a single file.

Here’s what the contents of the instructions file in the example above looks like:

{
    "o": {"size": "full", "format": "", "rotation": "-1" },
    "b": {"size": "!2048,1536", "format": "jpg" },
    "d": {"size": "full", "quality": "dither", "region": "-1,-1,320,320", "format": "jpg" }	
}

This tells iiif-process to create a copy of the original file (o), a scaled version with maximum dimensions of 2048 or 1536 pixels (b) and a square-cropped thumbnail 320 pixels to a side that has been halftoned (d).

Strictly speaking these are go-iiif image request parameters since non-standard flags like quality=dither or rotation=-1 are accepted. These were introduced in version 1.1 of go-iif and are discussed in detail in the Using IIIF at SFO Museum blog post.

Here’s the final output of the iiif-process process:

{
  "source/IMG_0084.JPG": {
    "dimensions": {
      "b": [
        2048,
        1536
      ],
      "d": [
        320,
        320
      ],
      "o": [
        4032,
        3024
      ]
    },
    "palette": [
      {
        "name": "#b87531",
        "hex": "#b87531",
        "reference": "vibrant"
      },
      {
        "name": "#805830",
        "hex": "#805830",
        "reference": "vibrant"
      },
      {
        "name": "#7a7a82",
        "hex": "#7a7a82",
        "reference": "vibrant"
      },
      {
        "name": "#c7c3b3",
        "hex": "#c7c3b3",
        "reference": "vibrant"
      },
      {
        "name": "#5c493a",
        "hex": "#5c493a",
        "reference": "vibrant"
      }
    ],
    "uris": {
      "b": "source/IMG_0084.JPG/full/!2048,1536/0/color.jpg",
      "d": "source/IMG_0084.JPG/-1,-1,320,320/full/0/dither.jpg",
      "o": "source/IMG_0084.JPG/full/full/-1/color.jpg"
    }
  }
}

Every response will contain a key whose value is the name of the image you asked to process and whose value will be a dictionary with at least two keys: dimensions and uris.

These represent the sizes and paths for the each of the derivative images and their values are also dictionaries whose own keys map to the keys defined in your “instructions” file.

In this example we’ve also configured go-iiif to extract colour information for the source image so that information is encoded in the palette response key.

What you don’t see in these examples is that all three of the derivative images (b, d and o) were processed and stored concurrently, taking advantage of all the processors that a computer running iiif-process has at its disposal.

Amazon, for example, has lots and lots and lots of processors at its disposal. The good news is that we can. The bad news is that doing so is a bit like putting on a snow suit to go outside in the winter; there are many steps, it’s a bit awkward and the closer you get to reaching the outdoors the more uncomfortable it becomes.

Processing images as ECS tasks, from the command line

As mentioned earlier there is a second tool called iiif-process-ecs. It is a helper-tool designed to invoke a copy of iiif-process that’s been embedded in a Docker container along with a configuration file and an instructions file, and that is run as a “task” on Amazon’s servers.

Remember, Dockerfiles allow you to create bespoke shrink-wrapped virtual computers and ECS will inflate those “computers” on demand, run software installed on them and then shut everything down. The details of creating a iiif-process Dockerfile and of configuring it for use in AWS are covered in the go-iiif-aws documentation but the result is a container with both software and configuration files pre-installed.

In order to process the same source/IMG_0084.JPG in the example above as an ECS task you would invoke the iiif-process-ecs tool like this:

$> iiif-process-ecs -mode task \
   -ecs-dsn 'region={AWS_REGION} credentials={AWS_CREDENTIALS}' \
   -subnet {AWS_SUBNET} \
   -security-group {AWS_SECURITY_GROUP} \
   -cluster iiif-process-ecs -container iiif-process-ecs \
   -task iiif-process-ecs:1 \
   -uri source/IMG_0084.JPG

And the output would be something like this:

2019/01/30 15:30:01 arn:aws:ecs:{AWS_REGION}:{AWS_ACCOUNT_ID}:task/{ECS_TASK_ID}

As of this writing the iiif-process-ecs exits as soon as the ECS task is registered but not necessarily completed. It is left up to you ask AWS whether the task is complete. This isn’t ideal and one reason that the go-iiif-aws package is only considered to be a 0.0.1 release.

The other thing you might notice is that there is no output like there is in the command-line example. This is also on the “to do” list and wrapped up in the details of an entirely other AWS service called CloudWatch (vendor lock-in, anyone?) that manages when and where the output of an ECS task is recorded.

The normal iiif-process output will eventually be sent to CloudWatch but we haven’t written the code to extract it yet. We’d love some help if you are comfortable writing code in Go.

The alternative is to invoke the iiif-process-ecs tool with the -report flag. This flag will get passed down to the iiif-process tool which instructs the code to store a JSON-encoded copy of the response details to the same location as your derivative images.

Here’s an abbreviated example:

$> iiif-process-ecs -mode task \
   ... 
   -report \
   -report-name process.json \
   -uri source/IMG_0084.JPG

Which would cause the following files to be created:

source/IMG_0084.JPG/full/!2048,1536/0/color.jpg
source/IMG_0084.JPG/-1,-1,320,320/full/0/dither.jpg
source/IMG_0084.JPG/full/full/-1/color.jpg
source/IMG_0084.JPG/process.json

Again, this is not ideal but it does prevent AWS-specific infrastructure from worming its way in to, and becoming a dependency of, the core go-iiif codebase. In the short-term it is understood to be an acceptable trade-off.

Processing images as ECS tasks, from S3

You can also set up the iiif-process-ecs tool to run as an AWS Lambda function to be triggered when someone uploads a new file to an S3 bucket. The details of setting up a iiif-process-ecs Lambda function are best left to the go-iiif-aws documentation but the end result is that you are basically asking Amazon to do this (again, abbreviated) when someone uploads new file:

$> iiif-process-ecs -mode task \
   ... 
   -uri FILE_THAT_JUST_GOT_UPLOADED_TO_S3

What happens after the iiif-process task is completed is outside the scope of the Lambda function. You will need to account for that in a separate piece of code. Because the Lambda function is just invoking the ECS task all the caveats about return values, discussed above, apply here too.

Magic, right? Not quite, if we’re being honest (or if you’re the person who has to set this up) but it’s a start.

Wrapping it up (for now)

The actual iiif-process tool is part of the main go-iiif package, which itself has been moved in to a newly created go-iiif organization on GitHub. If you are already using go-iiif in your code you will need to update your import paths accordingly.

All of the AWS specific code, including Dockerfiles, is part of a standalone go-iiif-aws package.

There’s also another go-iiif-uri package that the first two use. There is also a dedicated package for URIs because even though we are using go-iiif to process images we don’t want our final images URLs to look like this:

source/IMG_0084.JPG/-1,-1,320,320/full/0/dither.jpg.

We want them to look like this, instead:

137/689/158/3/1376891583_MfZTKW2yv1csD2j6Xb53m7UCtHkC8YFi_sd.jpg

We use the go-iiif-uri package to accomplish this, but we’ll save that for a future blog post.

Everything described in this blog post is code we’ve begun using in earnest to start working through the backfill of installation photos for exhibitions. For any given image we generate (10) derivative images.

{
    "o": {"size": "full", "format": "", "rotation": "-1" },
    "k": {"size": "!2048,1536", "format": "jpg" },
    "b": {"size": "!1024,768", "format": "jpg" },
    "c": {"size": "!800,600", "format": "jpg" },
    "dc": {"size": "!800,600", "quality": "dither", "format": "jpg" },
    "z": {"size": "!640,480", "format": "jpg" },
    "n": {"size": "!320,240", "format": "jpg" },
    "s": {"size": "full", "region": "-1,-1,320,320", "format": "jpg" },
    "ds": {"size": "full", "quality": "dither", "region": "-1,-1,320,320", "format": "jpg" }
}

All the images for the recent Reflections in Wood – Surfboards & Shapers, The Modern Consumer – 1950s Products and Styles and A Sterling Renaissance, British Silver Design 1957-2018 exhibitions have been generated using this code in a fraction of the time (and a fraction of the cost) that doing so on a standalone server would have taken.

Our overall image processing workflow is more complicated than the iiif-process tool alone, and we are still working out the kinks, but it is the foundational layer for everything else we’re doing. We also think this is work that can be used to improve the time it takes to generate the tiles that power zoomable image interfaces.

We’re pleased to make our contributions available to others in the sector and we hope that others will find them as useful as we do.