lucasjinreal/Crane

🚀 What is Crane?

Crane (Candle‑based Rust Accelerated Neural Engine) is an open‑source inference framework written in Rust that builds on the Candle machine‑learning library. Its goal is to make large‑language‑model (LLM) and multimodal model inference fast, portable, and easy to use on a variety of hardware – CPU, NVIDIA CUDA, Apple Metal, and even AMD ROCm.

Core ideas

Idea	What it means
Rust + Candle	Rust gives safety and a single‑binary deployment model; Candle provides PyTorch‑like APIs and a set of optimized kernels written in Rust/CUDA/Metal.
GGUF‑first	Models are loaded directly from the GGUF format (the same format used by llama.cpp). No extra conversion steps are required.
Hardware‑agnostic back‑ends	At runtime the same model can run on CPU, CUDA, Metal (macOS) or ROCm (AMD) without changing the code.
OpenAI‑compatible server	crane‑serve implements the OpenAI chat/completions API (and SGLang), so existing client code works out of the box.
Out‑of‑box AI abilities	Besides plain text generation it ships ready‑to‑use pipelines for OCR, speech‑to‑text (ASR), text‑to‑speech (TTS), music transcription, and vision‑language tasks.

What you can do with it

• Run huge models on modest hardware – e.g., the 27 B Bonsai ternary model fits in 16 GB of RAM and runs on CPU or a single GPU.
• Serve models via an API – start crane‑serve and call /v1/chat/completions, /v1/audio/speech, etc.
• Write your own Rust app – the crane-core crate exposes a clean Rust API (ModelForCausalLM, GenerationConfig, tokenizers, etc.) so you can embed inference in any Rust binary.
• Experiment with quantization – --quant q4_0 and similar flags perform in‑situ quantization while loading, reducing VRAM/RAM usage.

📦 Supported models (as of the README)

• Bonsai 2 Ternary 27B – native Prism PTQ1_0 / PQ2_0 GGUF, CPU & CUDA.
• Qwen 3.8‑Flash‑Next (177 B MoE) – runs on an Intel Arc GPU via SYCL.
• Qwen 3.5 / 3.6 / 3.8 (various sizes, dense and vision‑language variants).
• Ornith‑1.0‑9B – tool‑calling agent.
• Gemma 4, Hunyuan Dense, Moonshine ASR, Silero VAD, Qwen3‑TTS, Voxtral‑4B‑TTS, KugelAudio, MuScriptor (music transcription), PaddleOCR‑VL, etc.
• Many more are listed as “to‑come”.

🛠️ Getting started (quick‑start)

# clone & install
git clone https://github.com/lucasjinreal/Crane
./install.sh

# run the server (example with the Bonsai ternary model)
./target/release/crane-serve \
    --model-path checkpoints/Ternary-Bonsai-2-27B-gguf/Ternary-Bonsai-2-27B-PTQ1_0.gguf \
    --ui

The UI opens in a browser and lets you chat, run OCR, ASR, etc., without writing code.

Running a simple Rust program

The README includes a minimal example that loads Qwen2.5‑0.5B‑Instruct and generates a response in pure Rust – only a few dozen lines of code.

⚡ Performance highlights (from the changelog)

Hardware	Model	Speed vs. PyTorch / llama.cpp
Apple M1/M2/M3 (CPU)	Qwen3‑VL 2B	~50× faster than native PyTorch
NVIDIA RTX 3090 (CUDA)	Qwen 3.5‑2B‑Q8_0	3× faster than llama.cpp (after kernel‑fusion work)
AMD RX 7800 XT (ROCm)	Qwen 3.5‑2B‑Q8_0	5–7× faster than CPU, closing the gap to CUDA
Intel Arc Pro B70 (SYCL)	Qwen 3.8‑Flash‑Next	runs on a single 32 GB GPU

Key optimisations include:

• Kernel fusion (norms, GDN recurrence, top‑k sampling).
• In‑situ quantization that avoids loading full‑precision weights into GPU memory.
• Reduced kernel‑launch overhead – submission time cut from ~22 ms to ~7 ms per token.

📚 How it works under the hood

1. Model loading – crane-core reads the GGUF header, detects the architecture (e.g., qwen3_5), and builds a Candle VarBuilder with the appropriate dtype/device.
2. Backend selection – At runtime the device (cpu, cuda, metal, rocm) is chosen; the same Rust code dispatches to the matching Candle kernels.
3. Generation loop – Uses Candle’s ModelForCausalLM trait; a TextStreamer streams tokens back to the client.
4. Server layer – crane-serve wraps the generation logic in an async HTTP server exposing OpenAI‑compatible endpoints.

🛣️ Roadmap / What’s next

• More multimodal models (VLA, advanced vision‑language pipelines).
• Full‑fledged MoE support on ROCm.
• Better memory‑efficient attention kernels (flash‑style) to remove the score matrix.
• Expanded tooling for model conversion and quantization.

---

All information above is taken directly from the repository’s README; no external assumptions have been added.