Michael-A-Kuykendall/shimmy

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

What it solves

Shimmy provides a lightweight, single-binary inference server that allows users to run GGUF models locally and privately. It eliminates the need for heavy dependencies like Python runtimes or C++ toolchains, offering a fast-starting, low-memory alternative to tools like Ollama.

How it works

Shimmy acts as an OpenAI-compatible API server. It uses the Airframe engine—a pure-Rust transformer engine that leverages WebGPU (WGSL) compute shaders to run models on various GPUs (NVIDIA, AMD, Intel, Apple Silicon). It automatically derives model specifications from GGUF metadata, ensuring deterministic output through F32 accumulation precision.

Who it’s for

It is designed for developers and users who want a minimal, high-performance local LLM server that integrates seamlessly with existing AI tools via the OpenAI SDK surface.

Highlights

  • OpenAI Compatibility: Supports chat completions, text completions, and streaming.
  • Pure Rust: 100% Rust implementation with zero dependencies and a <1s startup time.
  • WebGPU Acceleration: Cross-platform GPU support via WGSL compute shaders.
  • TurboShimmy: An INT4 KV cache compression feature that significantly reduces VRAM usage.
  • Certified Models: A rigorous 3-box certification regimen (MATH, INFERENCE, DETERMINISM) for supported model/quant combinations.
  • Extended Context: Supports YaRN RoPE scaling for expanded context windows.

Related

  • Project
    guoqingbao/xinfer

    A blazing-fast LLM inference engine written in pure Rust that eliminates Python/PyTorch dependencies to enable high-throughput, production-ready model serving on consumer and professional GPUs.

  • Project
    EricLBuehler/mistral.rs

    A fast, flexible LLM inference engine written in Rust that supports multimodal models, automatic quantization, and a built-in agentic runtime for tool and code execution.

  • Project
    pegainfer-project/pegainfer

    A pure Rust and CUDA LLM inference engine that removes framework runtimes like PyTorch to provide faster startup times and a smaller memory footprint.

  • Project
    pykeio/ort

    A Rust interface for hardware-accelerated inference and training of ONNX machine learning models, enabling efficient deployment across various hardware and platforms.

  • Project
    Avarok-Cybersecurity/atlas

    A pure Rust LLM inference engine that uses hardware-specific kernels to maximize local inference speed and stability.