WallLabel – Experiments with Apple's open source machine-learning frameworks

At the end of the blog post Registrar – Experiments with Apple’s on-device machine-learning frameworks, about Apple’s newly available on-device machine-learning FoundationModel framework to derive structured data from wall label text, I wrote:

It stands to reason that in some future release the Apple FoundationModel framework will support the use of custom models for performing tasks but until then we’ll keep chipping away the code to switch between using the on-device models and bespoke models installed locally and manipulated using the llama.cpp Swift bindings. While the built-in functionality that Apple has made available by default is certainly convenient it seems expedient to have another way, where we can exert a modicum of control, to do the same things.

After that post was published I was reminded that Apple has also been working on a separate, open source framework, called MLX, for manipulating third-party machine-learning models and that it has Swift bindings for developing native applications. This blog post is about the work we’ve done to add support for the MLX bindings to the Registrar application allowing it to use both the built-in FoundationModel framework, if available, as well as third-party models. This enables the Registrar application to work on devices not running AppleOS 26 or which have not enabled the system-wide Apple Intelligence setting.

All of this work has been bundled in to its own Swift package, named WallLabel, with a simple-to-use API. For example:

import WallLabel

let parser_uri = "mlx://?model=llama3.2:1b"
let label_text = "YOUR LABEL TEXT HERE"

var label_parser: Parser
        
do {
	label_parser = try NewParser(parser_uri: parser_uri, logger: nil)
} catch {
	// throw error here...
}
        
let parse_rsp = await label_parser.parse(text: label_text)
        
switch parse_rsp {
case .success(let label):
	// do something with label here
case .failure(let error):
	// throw error here
}

The WallLabel packages exposes a Parser instance for deriving structured data from freeform wall label text. These parsers are defined using a URI-based syntax which encodes all of the details specific to that parser. For example, the default parser which uses the built-in FoundationModel framework is defined by the URI foundation://. Parsers which use third-party models and the MLX framework are defined by mlx://?model={MODEL_NAME} URIs. Eventually there will be a parser to support using the llama.cpp bindings and its URI will be llama:// and so on. The default parser for the Registrar application continues to be the built-in FoundationModel but custom models can be configured in the application’s Settings panel.

Putting all of this logic in to its own package also means its possible to write a standalone command-line tool for parsing label text and testing models independent of all the other things that the Registrar application does. For example, here’s how you would parse label text using the llama3.2:1b model:

$> ./bin/wall-label parse \
   	--parser_uri 'mlx://?model=llama3.2:1b' \
	--label_text 'Promotion, Chiat/ Day: Effective Brick Design Director: Tibor Kalman (American, b. Hungary, 1949–1999); Firm: M&Co (United States); USA offset lithography Gift of Tibor Kalman/ M & Co. Cooper Hewitt Smithsonian National Design Museum 1993-151-257-1' \

| jq

{
  "longitude": 0,
  "input": "Promotion, Chiat/ Day: Effective Brick Design Director: Tibor Kalman (American, b. Hungary, 1949–1999); Firm: M&Co (United States); USA offset lithography Gift of Tibor Kalman/ M & Co. Cooper Hewitt Smithsonian National Design Museum 1993-151-257-1",
  "date": "1993",
  "creator": "Tibor Kalman (American, b. Hungary, 1949–1999)",
  "creditline": "Cooper Hewitt Smithsonian National Design Museum",
  "accession_number": "151-257-1",
  "timestamp": 1761336687,
  "latitude": 0,
  "location": "1993-151-257-1",
  "title": "Effective Brick",
  "medium": "offset lithography"
}

One of the notable differences between the AppleOS 26 FoundationModel and the MLX frameworks is that the latter makes a good faith effort to generate structured data but can not yet guarantee it by default the way the former does. For example, some models, like qwen2.5:1.5b, will return structured data encoded as JSON but also wrapped in Markdown:

$> bin/wall-label parse \
	--parser_uri 'mlx://?model=qwen2.5:1.5b` \
	--verbose=true \       
	--label_text 'Promotion, Chiat/ Day: Effective Brick Design Director: Tibor Kalman (American, b. Hungary, 1949–1999); Firm: M&Co (United States); USA offset lithography Gift of Tibor Kalman/ M & Co. Cooper Hewitt Smithsonian National Design Museum 1993-151-257-1' \

| jq

2025-10-24T13:14:17-0700 debug org.sfomuseum.wall-label: [WallLabel] Load model qwen2.5:1.5b
2025-10-24T13:14:17-0700 debug org.sfomuseum.wall-label: [WallLabel] Load model (qwen2.5:1.5b) from source
2025-10-24T13:14:23-0700 debug org.sfomuseum.wall-label: [WallLabel] Loading model qwen2.5:1.5b 100.0% completed
2025-10-24T13:14:30-0700 debug org.sfomuseum.wall-label: [WallLabel] INFO GenerateCompletionInfo(promptTokenCount: 404, generationTokenCount: 105, promptTime: 0.22025799751281738, generateTime: 1.4748320579528809)
2025-10-24T13:14:30-0700 debug org.sfomuseum.wall-label: [WallLabel] DONE ```json
{
  "title": "Promotion",
  "date": "",
  "creator": "Chiat/ Day",
  "creditline": "Director: Tibor Kalman (American, b. Hungary, 1949–1999); Firm: M&Co (United States)",
  "location": "",
  "medium": "",
  "accession_number": "1993-151-257-1",
  "input": ""
}
```

Since the MLX frameworks do not guarantee structured output the text returned by the model needs to be parsed as JSON which, like in the example above, might fail. In many ways the WallLabel package is little more than one-among-many large language model “runners”. These are all opinionated abstractions, or wrappers, around a core library for performing the number-crunching that make large language models work and are generally designed to provide a simplified interface, often web or API based, for use in a variety of environments. The WallLabel is no different except that it attempts to return a wall label specific data type:

public struct WallLabel {
    /// The title of the object
    var title: String?
    /// The date attributed to an object, typically when that object was created
    var date: String?
    /// The individual or organization responsible for creating an object.
    var creator: String?
    /// The name of an individual, persons or organization who donated or are lending an object.
    var creditline: String?
    /// The location that an object was produced in.
    var location: String?
    /// The medium or media used to create the object.
    var medium: String?
    /// The unique identifier for an object.
    var accession_number: String?
    /// The Unix timestamp when a photo of an object was captured.
    var timestamp: Int?
    /// The latitude location coordinate of the camera capturing a photo of an object.
    var latitude: Float64?
    /// The longitude location coordinate of the camera capturing a photo of an object.
    var longitude: Float64?
}    

There are some ongoing efforts to ensure structured data output when using the Swift MLX bindings but I have not had time to test them yet so the issues described above, along with the occassional crazy-talk that any given model will produce, continue to be an avenues of investigation and improvement. Suggestions and contributions are, as always, welcome. The code is available from our GitHub account:

The WallLabel package tries to follow the idea of being a “small focused tool”. It only does one thing (maybe two) and definitely does not solve the myriad of issues – technical, social and financial – that machine-learning and “AI” systems raise. This is especially true for museums which often lack the bandwidth to address the enormity of the possibility space and the quicksand of a potentially one-sided future being marketed, by some, as an inevitability.

On-device models are still someone else’s models but having the flexibility to choose one model over another, to recognize that they are systems with strengths and weaknesses rather than all-knowing oracles, and the ability to incorporate those choices in to how our projects are designed and implemented is a small, but important, step in retaining some degree of control and agency in our work. To quote Paul Ford, from his essay The Argument for Letting AI Burn It All Down:

We have to learn how to teach people about LLMs, about how to put guardrails on AI projects and not just count on OpenAI to do it for us. We’ll have to ship products, and make smarter chatbots, and help people use these tools in good ways, even as other people are using ChatGPT to automate their dating lives. We’ll have to figure out where vibe coding is good and where it’s dangerous. And we’ll have to do it all while the world melts, both cognitively and glacier-ly, knowing that AI is contributing to that melt. On one side you’ve got these new human-impersonating machines that spew words, images, videos, and sounds by the millions per minute, and on the other you’ve got 8 billion agitated primates with access to weapons. The entire human commons is about to become a Superfund site, and the people who made the mess will move on to quantum computing. Once the frenzy fades, there’s just going to be a lot to do, and less sovereign wealth with which to do it.

This is the work. Start where you are and share what you learn even if it doesn’t seem “important”. Everyone is busy and overwhelmed which means the small things often matter more than you might think.