Swift Transformers Release: Run On-Device LLMs on Apple Devices

Released Today

Hugging Face released swift-transformers, swift-chat, updated exporters, an updated transformers-to-coreml Space, and Core ML versions of Llama 2 7B and Falcon 7B to enable on-device LLMs on Apple devices. The swift-transformers package is an in‑progress Swift library that aims to provide a transformers‑like API for text generation. The swift‑chat app demonstrates how to use it. The exporters Python package and the transformers-to-coreml Space have been updated to convert models such as Llama 2 and Falcon to Core ML. Ready‑to‑use Core ML models, including Llama‑2-7b-chat-coreml and Falcon-7b-instruct, are available on the Hub.

Tasks Overview

Running an LLM on Apple devices involves model conversion to Core ML, optimization for speed and memory, and providing a Swift‑friendly API for tokenization, model loading, and text generation. Conversion is the first step, but optimization and a usable Swift interface are equally important. The post walks through each of these areas, noting where tools already exist and where work is still needed.

Conversion to Core ML

Converting models such as Llama 2 to Core ML can be done with the transformers-to-coreml Space, the exporters Python package, or directly with coremltools, and the process succeeded after fixing a type‑mismatch issue in the Llama 2 code. The recommended approach is to try the no‑code transformers-to-coreml Space first; if it fails, use exporters for more control, and fall back to coremltools for maximum flexibility. For Llama 2, the conversion worked with all three methods after addressing an earlier failure.

Important lessons learned

If you have to use coremltools, use the latest version: 7.0b1. Despite being a beta, it is stable, includes fixes, supports PyTorch 2, and offers advanced quantization tools. exporters no longer applies a softmax to outputs when converting text generation tasks, because this was necessary for some generation algorithms. exporters now defaults to using fixed sequence lengths for text models. Flexible shapes only run on CPU, not on GPU or the Neural Engine, so fixed shapes are required for GPU/Neural Engine execution.

Optimization

Key optimization strategies for Core ML LLMs include caching key‑value pairs, using discrete fixed shapes instead of flexible inputs, and applying quantization techniques such as 6‑bit palettization or mixed‑bit quantization. Without caching, the model recomputes attention scores for the full history each step. Using discrete shapes avoids the CPU‑only limitation of flexible inputs and, combined with caching, enables larger context sizes. Quantization reduces model size and resource usage; 6‑bit palettization and mixed‑bit quantization are highlighted as promising approaches.

swift-transformers

The swift-transformers package provides a Swift‑native tokenizers module, Hub‑aware model wrappers, and basic generation algorithms (greedy and top‑k sampling) for a growing set of models.

Tokenizers

Tokenization converts text to model‑ready IDs and back; swift-transformers currently supports BPE tokenizers used by GPT, Falcon, and Llama models, with code that mirrors the Hugging Face tokenizers library. Example usage:

import Tokenizers
func testTokenizer() async throws {
    let tokenizer = try await AutoTokenizer.from(pretrained: "pcuenq/Llama-2-7b-chat-coreml
    let inputIds = tokenizer("Today she took a train to the West
    assert(inputIds == [1, 20628, 1183, 3614, 263, 7945, 304, 278, 3122])
}

Model and Hub wrappers

A simple LanguageModel protocol wraps a Core ML model for text generation. The Hub module downloads tokenizer and configuration files from the Hub using custom metadata fields added to the Core ML model during conversion (automatically added by exporters and the Space).

Generation Algorithms

Two decoding strategies are implemented: greedy decoding (always picks the highest‑probability token) and top‑k sampling (randomly samples from the k most likely tokens, with temperature controlling variability). Additional methods such as nucleus sampling are planned.

Supported Models

So far the package has been tested with Llama 2, Falcon, StarCoder (a GPT‑variant), and the GPT family including GPT2, distilgpt, GPT‑NeoX, and GPT‑J.

swift-chat

The swift-chat demo app shows how to integrate swift-transformers in a Swift project, loading a Core ML model from the Hub and running text generation with automatic metadata download. After selecting a model, the app downloads the needed configuration files, compiles the model via Core ML (caching the result), and runs generation. The UI is intentionally simple and does not yet support features such as system prompts.

Missing Parts / Coming Next

Future work will add encoder‑decoder model support, Unigram and WordPiece tokenizers, additional generation strategies, key‑value caching, and discrete shape conversion for larger contexts. The team invites contributions via issues, pull requests, or feedback on the repositories.

Conclusion

These tools give Swift developers a starting point for bringing on‑device LLMs to Apple apps, and the team invites community contributions to improve them.

Sources