apple/foundation-models-utilities

Emerging and experimental patterns for building with the Foundation Models framework

What it solves

This package provides a set of utilities to simplify the integration and management of Large Language Models (LLMs) within the Apple Foundation Models framework. It specifically addresses the challenges of managing long conversation histories, connecting to various hosted model servers, and adding task-specific instructions without cluttering the model's context window.

How it works

The library offers three primary components:

  • ChatCompletionsLanguageModel: A client that allows developers to connect to any server implementing the chat completions REST API, making it compatible with a wide range of open-source LLM utilities.
  • History Management: A suite of profile modifiers that prevent context window overflow by compressing transcripts. This is achieved through strategies like rolling windows, summarizing previous interactions, or dropping completed tool calls.
  • Skills: A system for adding just-in-time directions to a session. Skills can be added as prompts (which avoid invalidating the key-value cache) or as instructions (which are high priority but may invalidate the cache). The model can activate these skills via tool calls, and instructions-based skills can be optionally deactivated to further reduce context pollution.

Who it’s for

Swift developers building AI-powered applications on Apple platforms or supported Linux distributions who are using the Foundation Models framework.

Highlights

  • REST API Compatibility: Connects to any chat completions compatible server.
  • Flexible History Compression: Composable strategies for rolling windows and summarization.
  • Dynamic Context Management: Just-in-time "Skills" to optimize time-to-first-token and reduce context pollution.
  • Cross-Platform Support: Works on Apple platforms and select Linux distributions like Ubuntu.

Related

  • Project
    eastriverlee/LLM.swift

    A Swift library for running local LLMs on Apple platforms, providing high-level abstractions for structured output, tool use, and chain-of-thought reasoning.

  • Project
    ml-explore/mlx-swift-lm

    A Swift package for building applications with large language models and vision language models using MLX Swift, featuring model loading, fine-tuning, and guided generation.

  • Project
    simonw/llm

    A CLI tool and Python library for interacting with multiple LLMs, both via remote APIs and local installations, featuring built-in logging and structured data extraction.

  • Project
    tidyverse/ellmer

    A package that makes it easy to use large language models from R, supporting a wide range of providers and features like streaming and tool calling.

  • Project
    alexzhang13/rlm

    An inference engine and training environment for Recursive Language Models (RLMs) that enable LLMs to handle near-infinite context by programmatically decomposing tasks via recursive sub-calls in a REPL environment.