magnitudedev/magnitude

Open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. 2x faster than Ollama. Works on Apple Silicon, NVIDIA, AMD, or just a CPU.

What it solves

Magnitude addresses the performance gap in local LLM inference by moving away from precompiled kernels. Instead of using general-purpose kernels that work across broad hardware classes, it optimizes the model's execution for the specific chip on the user's device to maximize speed and memory efficiency.

How it works

Magnitude is an inference engine that compiles and tunes its kernels directly on the user's hardware before a model runs. It provides hand-optimized kernels for popular open-weight model families and utilizes shared prefix caches to maintain speed during concurrent sessions. It is delivered as a desktop application with an integrated CLI.

Who it’s for

Developers and users who want to run open-weight models locally on macOS, Windows, or Linux using Apple Silicon, NVIDIA, AMD GPUs, or CPUs, and who want to integrate these models with existing AI agents like Cline, Claude Code, or Pi.

Highlights

  • Hardware-specific tuning: Compiles and tunes kernels on-device for better performance than generalist engines.
  • Crosspatform support: Works across Apple Silicon, NVIDIA, AMD, and CPU-only setups.
  • Performance gains: Claims up to 2x faster performance than llama.cpp, specifically citing 92% faster decode on Metal and 19% on CUDA.
  • Agent integration: One-click connectivity for various agents and an OpenAI-compatible API for others.
  • Resource efficiency: Uses 27% less memory per agent and frees memory when agents are inactive.

Related

  • Project
    AlexsJones/llmfit

    A hardware-aware recommendation tool that analyzes your system specs to find the best-fitting open-source LLMs across various quantizations.

  • Dispatch
    Magnitude: Self-Optimizing Inference Engine for Local AI Agents

    Magnitude is an open-source inference engine that compiles and tunes kernels on-device to achieve up to 2x faster performance than llama.cpp for local AI agents.

  • Project
    midudev/canirun.ai

    A hardware detection and recommendation tool that tells users which open-weight AI models will run best on their specific machine based on their CPU, RAM, and GPU.

  • Project
    signerless/llm-checker

    An AI-powered CLI tool that analyzes hardware and recommends optimal local LLM models from a multi-source registry, featuring integrated safety verification and MCP support.

  • Project
    InfiniTensor/InfiniTensor

    InfiniTensor is an open‑source C++ runtime that imports ONNX graphs, applies generic graph rewrites, and executes them on CPU or on custom accelerator back‑ends (InfiniOps/InfiniRT). It offers Python bindings, a test suite, and a modular design that lets hardware vendors plug in their own kernels.