ANEMLL – Running Large Language Models on Apple’s Neural Engine

What it is – ANEMLL (pronounced “animal”) is an open‑source toolkit that lets you take LLMs from Hugging Face and run them locally on Apple Silicon devices (iPhone, iPad, Mac, visionOS) using the Apple Neural Engine (ANE). It covers the whole pipeline: model conversion to Core ML, profiling tools, Swift‑based reference apps, and Python‑level chat demos.

---

Core components (as described in the README)

Component	What it does
LLM Conversion Tools	Scripts (convert_model.sh, convert_monolith.sh) that ingest Hugging Face weights and emit Core ML models (single‑file monolithic or chunked). Supports LLaMA, Qwen, Gemma 3, DeepSeek, etc., with optional in‑model argmax and weight deduplication (ANEMLL‑Dedup).
ANE Profiler	A Core ML/ANE profiler that works without Xcode, giving compute‑plan breakdowns, compatibility reports and benchmark numbers.
Swift reference implementation	A Swift library and CLI (anemll‑swift‑cli) that demonstrates fast on‑device inference, plus a full‑screen chat UI for iOS/macOS/visionOS (voice input, AirDrop model sharing, Markdown rendering).
Python sample code	chat.py (quick test) and chat_full.py (conversation‑history handling) plus a suite of conversion‑validation tests.
iOS/macOS sample apps	A redesigned “ANEMLL Chat” app (available via TestFlight) that showcases end‑to‑end model download, local import, and streaming chat on device.
ANEMLL‑BENCH	Separate repo for systematic ANE performance benchmarking across models and quantization levels.

---

Supported models (stable as of v0.3.5)

• Gemma 3 – 270 M, 1 B, 4 B (quant‑aware‑training) with sliding‑window + global attention, up to 4 K context.
• LLaMA 3.1/3.2 – 1 B, 8 B.
• Qwen 3 – 0.6 B, 1.7 B, 8 B.
• Qwen 2.5 – 0.5 B, 1.5 B, 3 B, 7 B.
• DeepSeek R1 – 8 B distilled (LLaMA‑based).
• DeepHermes – 3 B, 8 B (fine‑tuned LLaMA).

All of these can be converted to a Core ML model that runs entirely on the ANE, with optional quantization (LUT4/LUT6) and a special in‑model argmax mode that reduces host‑device data transfer.

---

Highlights of the 0.3.5 Beta release

• Full‑stack iOS/macOS/visionOS chat app with voice input, AirDrop sharing, Markdown rendering, and a “thinking mode”.
• Monolithic model support – a single Core ML file per model, cutting storage overhead by ~50 % thanks to ANEMLL‑Dedup.
• In‑model argmax (--argmax) – moves the argmax operation into the Core ML LM head, sending only the winning token index/value back to the host, dramatically lowering bandwidth.
• Swift inference stability – IOSurface‑backed buffers and a serial prediction queue eliminate race conditions on iOS.
• ANE Profiler – command‑line profiling without Xcode, useful for spotting compatibility issues.
• Benchmark results – on several standard NLP tasks ANEMLL‑FP16 slightly outperforms Hugging Face FP16 on the same hardware (+0.71 % average accuracy across ARC, BoolQ, etc.).
• Automatic virtual‑environment activation – conversion scripts will spin up the env-anemll venv unless you disable it.

---

Typical workflow (quick‑start)

# 1. Create a reproducible Python env (uv is recommended)
brew install uv
./create_uv_env.sh               # makes env-anemll with Python 3.9
source env-anemll/bin/activate
./install_dependencies.sh       # coremltools, transformers, etc.

# 2. Convert a model (example: Gemma‑3‑270M)
./anemll/utils/convert_model.sh \
    --model google/gemma-3-270m-it \
    --output ./models/gemma3_270m \
    --context 512 \
    --chunk 1 \
    --argmax

# 3. Test inference in Python
python ./tests/chat.py \
    --meta ./models/gemma3_270m/meta.yaml \
    --prompt "Hello, ANEMLL!"

# 4. (Optional) Run the Swift CLI or the iOS Chat app
./anemll-swift-cli/run.sh ./models/gemma3_270m/meta.yaml
# or launch the TestFlight app for a full UI experience

The README also provides test scripts (tests/conv/*.sh) that automatically download tiny reference models, convert them, and run end‑to‑end validation.

---

Why you might use ANEMLL

• On‑device privacy – No network calls; the entire LLM runs locally on the device’s ANE.
• Low‑power edge AI – Suitable for mobile or embedded Apple‑Silicon products where battery life matters.
• Fast inference – ANE can execute FP16 matrix ops in parallel; the in‑model argmax further reduces latency.
• Developer-friendly – Comes with Python utilities for quick prototyping and a Swift library for production‑grade iOS/macOS apps.

---

Installation & system requirements (summarised)

• macOS Sequoia (or later) on Apple Silicon (M1/M2/… recommended).
• ≥ 16 GB RAM (32 GB for 8 B models).
• Python 3.9‑3.11 (3.9 is the most tested).
• Xcode command‑line tools (for the Core ML compiler).
• coremltools ≥ 9.0, transformers ≥ 4.36.0, plus NumPy, scikit‑learn, etc.
• Optional: uv for fast environment creation.

---

Where to find more

• Documentation – docs/ folder (conversion guide, profiler docs, Swift CLI guide, sample‑app instructions).
• Pre‑converted models – Hugging Face organization anemll (ready‑to‑use .mlmodelc bundles).
• Benchmarks – ANEMLL‑BENCH repo and the table of task‑level results in the README.
• Community – Follow @anemll on X/Twitter for updates; star the repo to show support.

---

TL;DR

ANEMLL is a genuine, actively maintained open‑source project that bridges the gap between large language models and Apple’s on‑device neural hardware. It supplies conversion scripts, profiling tools, Swift and Python reference implementations, and ready‑made iOS/macOS chat apps, enabling developers to run privacy‑preserving LLMs on iPhones, iPads, Macs, and visionOS devices.