magnitudedev/magnitude

Open source agent with local models built in. Fully private and offline. Works out of the box on any hardware.

What it solves

Magnitude removes the guesswork from running local AI models. Instead of users manually searching for compatible model sizes, quantization levels, and performance estimates, Magnitude automatically profiles the user's hardware to recommend and run the best-fitting models.

How it works

The system profiles the machine's chip, memory, and bandwidth to suggest models with estimated tokens per second. It functions as an open-source inference server that downloads, tunes, and runs models (including GGUF models from Hugging Face). It optimizes performance using speculative decoding and concurrency settings tailored to the hardware. To manage resources, models are loaded just-in-time upon request and unloaded when the system is idle or memory becomes limited.

Who it’s for

Developers building or using AI agents who want a fully private, offline, and free local inference setup without needing to deep technical knowledge of hardware compatibility.

Highlights

  • Hardware-aware recommendations: Automatically profiles hardware to suggest the best models and estimated speeds.
  • Agent-optimized: Specifically tuned for agent workloads with dynamic model loading and unloading.
  • Privacy-first: Operates entirely offline; all prompts, files, and models remain on the local machine.
  • Broad compatibility: Works with various harnesses like Pi, OpenCode, Hermes, and Cline.
  • One-command setup: Simplified installation and configuration via a CLI.

Related

  • Project
    signerless/llm-checker

    An AI-powered CLI tool that analyzes system hardware to recommend and verify the optimal local LLM models for performance and compatibility.

  • Project
    facebookresearch/mobile-vision

    A collection of hardware-aware efficient ConvNet models and tools, including FBNet and ChamNet, designed to optimize computer vision performance and latency on mobile devices.

  • Dispatch
    AI & Frontier Tech Roundup: Localized Frontier Models, Agentic Infrastructure, and Physical AI

    The AI landscape is shifting toward making frontier-class intelligence cheap and local, with breakthroughs in MoE inference on consumer hardware and a surge in agentic infrastructure and physical robotics.

  • Project
    InfiniTensor/InfiniTensor

    A high-performance inference engine for GPUs and AI accelerators that enables efficient model deployment and academic validation.

  • Dispatch
    Hugging Face Accelerated Inference API Optimization

    Hugging Face achieved a 100x speedup in transformer inference for its Accelerated Inference API by combining high-level library optimizations, Rust-based tokenization, and hardware-specific compilation.