lucasjinreal/Crane

Crane – a Rust‑based, Candle‑powered inference engine

What it is

• Crane (Candle‑based Rust Accelerated Neural Engine) is an open‑source framework for running large language models (LLMs) and multimodal models (vision, OCR, speech‑to‑text, text‑to‑speech, music transcription, etc.) at high speed.
• It is built on the Candle tensor library written in Rust, which provides fast, low‑level kernels for CPU, NVIDIA CUDA, Apple Metal, and experimental AMD ROCm back‑ends.
• The project ships three crates:
  1. crane‑core – the library that implements model loaders, tokenizers, generation logic and the custom kernels.
  2. crane – a collection of example binaries (chat, VLM, OCR, TTS, …) that show how to use the core library.
  3. crane‑serve – an OpenAI‑compatible HTTP server (also supports SGLang) that exposes chat, completion, audio‑speech, and other endpoints.

Key selling points (as described in the README)

• Speed – claims up to 50× faster than native PyTorch on Apple Silicon and several‑fold faster than llama.cpp on GPUs, thanks to fused kernels, KV‑cache quantisation, and kernel‑launch reduction.
• Rust‑only codebase – avoids C++ complexity while still delivering native performance.
• Cross‑platform – one code path works on CPUs, CUDA GPUs, Metal GPUs (macOS), and experimental ROCm GPUs.
• OpenAI‑compatible API – the server implements /v1/chat/completions, /v1/completions, /v1/audio/speech, tokenisation endpoints, and SGLang‑specific routes, so existing client libraries can talk to it without changes.
• Model support – out‑of‑the‑box support for many recent models, including Qwen 3.5/3.6/3.8 (up to 27 B), Qwen 2.5, Gemma 4, PaddleOCR‑VL, MuScriptor (music transcription), various TTS models, ASR, VAD, etc. Models are loaded directly from GGUF files or from safetensors checkpoints with on‑the‑fly quantisation (--quant q4k|q8_0|…).
• Easy to add new models – most models need fewer than 100 lines of Rust to plug in because the architecture is detected from the GGUF header.

Typical workflow

1. Install Rust (stable toolchain).
2. Build the desired binary with the appropriate feature flag:

   # CPU only
   cargo build --release
   # macOS Metal + Accelerate
   cargo build --release --features "metal,accelerate"
   # NVIDIA CUDA
   cargo build --release --features cuda
   # AMD ROCm (experimental)
   cargo build --release --features rocm
   

3. Download a model (e.g. Qwen2.5‑0.5B‑Instruct) via huggingface-cli or by placing a GGUF file in a folder.
4. Run a demo – e.g. a chat binary:

   cargo run --bin qwenchat --release
   

5. Or start the server:

   ./target/release/crane --model-path /path/to/model
   # then call it from Python, curl, or any OpenAI‑compatible SDK
   

Example Rust snippet (from the README) shows a minimal end‑to‑end chat using the Qwen2.5 model, demonstrating how to:

• Load a tokenizer and model.
• Prepare a prompt with apply_chat_template.
• Configure generation parameters (max tokens, temperature, top‑p, etc.).
• Stream the generated text back to the console.

Why someone might choose Crane

• If you are already comfortable with Rust and want a single‑language stack for inference.
• If you need fast inference on Apple Silicon without fiddling with GGUF conversion tools.
• If you want a drop‑in OpenAI‑compatible server that also serves vision, OCR, ASR, and TTS endpoints.
• If you are experimenting with new quantisation or custom kernels and prefer a Rust‑centric codebase.

Current limitations (as noted)

• The ROCm backend is experimental and only tested on dense models.
• Multi‑sequence batching is limited (max_concurrent=1 for some VLM back‑ends).
• Certain advanced features (e.g., flash‑style attention) are still TODO.

Bottom line Crane is a genuine, actively maintained AI inference framework focused on speed, portability, and a Rust‑first developer experience. It targets the same niche as llama.cpp but aims to be cleaner, easier to extend, and capable of handling a broader set of modalities beyond pure text.