iOS Multi-screen Starter Kit

Last week, I delivered a presentation at the MCN 2020 (virtual) conference. It was one of two talks for a session titled Bring Your Own Device and In-Hand Interactives: How Covid-19 Forced Innovation in Our Interactives. This is part of what I said:

Before COVID-19 SFO Museum was working on its own touch-based interactive application, to be deployed in the terminals. It’s a plain-vanilla web application that is run inside of a native iOS application running on an iPad and mirrored to an external monitor. This is a pretty common pattern for a lot of applications so we’re not doing anything revolutionary here.

We opted to wrap the web application in an iOS application, rather than using a cross-platform framework like Cordova or Electron, because there are still a few things that can only be done natively. In our case: Displaying, and syncing, different content from the same application on two different screens.

Here’s a simplified view of what that looks like. Do you notice the yellow box marked “javascript” in the middle? I’ll come back to that in a moment.

There are two important reasons for developing the application itself using web technologies. The first is the ease and speed of development and being able to test things in a web browser. The second is that we know that whatever else happens in Apple’s ecosystem our application will still work in a web browser. In fact the guts of this application are already part of our website.

These boxes, which are all the same web application, are really the only SFO Museum-specific parts of what we’ve been developing.

In iOS, there is a bridge for communicating between the web application and the native application using JavaScript. For example, the application I’ve been describing is also meant to run offline so we are using some native iOS functionality to make that possible.

Our goal is to treat this as a fast, cheap and reusable template for building interactive installations. I am fond of telling people that we have developed a system for interactive displays with minimal hardware and stable costs, that work both online and offline, and that we can deploy to any four-by-four foot space at the airport. The only question should be: What do we want the next interactive to do?

It’s worth mentioning that what’s interesting about this isn’t so much that it’s an iOS application or that it’s running on an iPad. There are specific reasons why we chose these platforms but the same thing could be done with others. What’s interesting to me is a model for developing sophisticated interactive applications and identifying the separation of concerns between the applications themselves and all the plumbing necessary to run them.

We’re planning on open-sourcing this work sometime next week. Today, we have made that work public with the release of the ios-multiscreen-starter application.

What does the ios-multiscreen-starter application do? Very little, in fact.

The application’s main screen (as in the iOS device the application is running on) has a button that, when pressed, causes a message to be printed on an external display connected to the “controller” iOS device. This is just enough to demonstrate how to load different content on different displays and to communicate between the two and between the iOS and web application layers.

Here’s a simplified diagram outlining how it works:

Here’s what’s going on, in very broad strokes:

• In the application’s ViewController methods are registered that can be invoked from the JavaScript files in a WKWebKitView instance.

• Those methods, and their values, are intercepted by the userContentController method and rebroadcast as NotificationCenter messages (to the iOS application).

• Observers for those messages are registered in other parts of the code. When triggered, they perform any additional work (that you define in code) and then relay the result back to the WKWebKitView instance, in the form of a JavaScript function call and value, using the evaluateJavaScript method.

In the case of the ios-multiscreen-starter application this workflow implements a simple bucket-brigade to relay messages from one screen to another. It is meant to be a working reference implementation for other more sophisticated applications. It is not designed to be an abstract container in to which a pre-existing web application is placed and run as-is. For a good example of what’s possible, developing a hybrid native iOS and web application, have a look at Apple’s Discover WKWebView enhancements video from WWDC 2020.

You don’t need to be an XCode or a Swift expert in order to use ios-multiscreen-starter but there is some expectation that you know the basics of how iOS applications work and can put together a (simple) project and compile it from scratch. The goal with the project is to simplify things to the point where most of your time is spent developing the web application at the core of an interactive display rather than the plumbing and scaffolding necessary to run that application on an iOS device.

At the beginning and the end of my talk at MCN, I said:

Historically, one of the things that has characterized software projects built to be used by more than a single institution [in the cultural heritage sector] is that they are overly broad. They try to do too much and account for too many customizations necessary to account for too many different institutions. They are too much of everything and not enough of anything. We need more small, focused tools that don’t try to be all things to all people, all the time. … We need to develop a new practice for the way we build the tools we use internally so that they might be split in to smaller pieces that can be shared by as many people as possible externally.

…

No one has the time or the capacity to support the full weight of anything other than their own museum or institution. But maybe we have just enough time to grow and nurture a common kit of parts that can be re-used and adapted by and for individual organizations. It may be too soon to imagine that we can make everything easy but maybe we can start to make more things at least possible.

As with the NFC Clock application, we are sharing what we have learned already in a spirit of generosity and with the hope that it will make things just a little bit easier for the next person who asks “How might we…?” We hope that ios-multiscreen-starter can be a useful first step for other institutions developing their own hybrid web/native applications and we welcome comments, suggestions and improvements.

https://github.com/sfomuseum/ios-multiscreen-starter