Michael-A-Kuykendall/shimmy

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

What it solves

Shimmy provides a lightweight, single-binary inference server that allows users to run GGUF models locally and privately. It eliminates the need for complex C++ toolchains or Python runtimes, offering a high-performance alternative to larger tools like Ollama.

How it works

Shimmy acts as an OpenAI-compatible API server. It uses the Airframe engine—a pure-Rust transformer engine utilizing WebGPU (WGSL) compute shaders—to execute models across various hardware (NVIDIA, AMD, Intel, Apple Silicon). It automatically derives model specifications from GGUF metadata and supports F32 accumulation precision for deterministic output.

Who it’s for

Developers and AI enthusiasts who want a minimal, zero-dependency Rust-based server to integrate local LLMs into existing tools via an OpenAI-compatible interface.

Highlights

  • OpenAI SDK Compatibility: Supports chat completions, text completions, and streaming.
  • Pure Rust: 100% Rust implementation with no Python or C++ dependencies.
  • TurboShimmy: An INT4 KV cache compression feature that significantly reduces VRAM usage.
  • Broad Hardware Support: Works across various GPUs via WebGPU.
  • Certified Models: Includes a rigorous 3-box certification regimen (MATH, INFERENCE, DETERMINISM) for supported model/quant combinations.

Related

  • Project
    guoqingbao/xinfer

    A blazing-fast LLM inference engine written in pure Rust that eliminates Python/PyTorch dependencies to enable high-throughput, production-ready model serving on consumer and professional GPUs.

  • Project
    lablup/mlxcel

    A high-performance native Rust runtime for LLM, VLM, and audio inference on Apple Silicon, CUDA, and ROCm, removing Python from the production request path.

  • Project
    pykeio/ort

    A Rust interface for hardware-accelerated inference and training of ONNX machine learning models, enabling efficient deployment across various hardware.

  • Project
    Avarok-Cybersecurity/atlas

    A pure Rust and CUDA LLM inference engine that provides a lightweight, high-throughput server for large models without requiring Python or PyTorch.

  • Project
    mlc-ai/web-llm

    A high-performance in-browser LLM inference engine that uses WebGPU to run language models locally without server-side processing.