NVIDIA/Personal-AI-Router

Router that virtually distributes inference across connected devices in the home.

NVIDIA Personal AI Router (PAIR)

What it is – PAIR is a desktop‑plus‑service application that lets a group of computers on the same local network share the load of running LLM inference. It discovers other PAIR nodes, keeps track of which inference engines (Ollama, LM Studio, llama.cpp) and models each node has, and then routes incoming API calls (OpenAI‑style, Anthropic‑style, or Ollama‑style) to the best‑suited machine.

Key ideas

  • Local‑only routing – All traffic stays inside your LAN; no cloud component is required.
  • Per‑request routing – Each request is sent to a single node that has the requested model and enough free GPU/memory. PAIR does not split a single model across GPUs or pool GPU memory.
  • Cross‑platform – Runs on Windows 11, Linux (deb package), and macOS (dmg). Nodes of any supported OS can be mixed in a cluster, and both x86‑64 and arm64 are supported (Windows‑ARM is experimental).
  • Engine management – PAIR can install, start, stop, and update the supported engines for you, and it tracks which models are available in each engine.
  • Multiple API flavours – Exposes local proxy endpoints that mimic the OpenAI, Anthropic Messages, and Ollama APIs, so existing client code works unchanged.

Quick start (desktop flow)

  1. Download the appropriate installer from the GitHub releases page (Windows .exe, Linux .deb, macOS .dmg).
  2. Install – run the installer; it sets up background services, firewall rules, and a desktop UI.
  3. Launch PAIR – open the app like any other program.
  4. Add an engine – In Engine settings click Install next to Ollama, LM Studio, or llama.cpp. PAIR downloads and configures the engine automatically.
  5. Add a model – Click Add model, pick a model from the built‑in list (e.g., qwen4:12b or any GGUF model from Hugging Face), and let the engine download it.
  6. Test – Use the built‑in Test button or a simple curl request:
    curl http://127.0.0.1:11434/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"qwen4:12b","messages":[{"role":"user","content":"What does a router do?"}]}'
    The response is a standard OpenAI‑style JSON payload.
  7. Add more nodes – On Settings → Cluster generate a six‑digit PIN on the first machine, enter it on the second, and repeat the engine/model steps there. PAIR will now route requests to whichever node has capacity.

Terminal interface (no GUI)

For headless or server machines, PAIR provides a terminal UI that starts the same background services and shows a full‑screen view of nodes, jobs, and logs. All configuration actions available in the desktop app can be performed from the terminal.

How routing works

  • PAIR continuously monitors each node’s GPU and memory usage.
  • When a request arrives, it looks for a node that:
    1. Has the requested model loaded (or can load it quickly).
    2. Has enough free resources for the inference.
    3. Is currently the least loaded according to the scheduler.
  • The request is forwarded to that node’s local engine via the appropriate API endpoint. The response is proxied back to the original client.

Supported engines & models

Engine What it provides Model format
Ollama Full‑stack server with OpenAI‑compatible API gguf and native Ollama formats
LM Studio GUI‑driven server, also OpenAI‑compatible Same as Ollama
llama.cpp Lightweight CPU/GPU inference for GGUF files GGUF (*.gguf)

Installation & upgrade

  • Windows/macOS – Double‑click the installer and follow prompts. The installer registers a privileged helper, adds firewall rules, and sets up automatic update checks (opt‑in via Settings → Service).
  • Linux – sudo apt install ./NVPAIR-Setup-*.deb. Updates are delivered as new .deb packages; the app can self‑update when you enable it.
  • From source – See docs/building.mdx for prerequisites and the cargo/npm commands needed to compile the services and UI.

Uninstall / reset

  • Windows: uninstall from Apps & features; the uninstaller optionally removes PAIR data.
  • Linux: sudo apt remove nvpair (keeps data) or sudo apt purge nvpair (removes data).
  • macOS: run the bundled uninstall script; add --purge to delete data.
  • Reset data – Settings → Service → Reset app data clears PAIR’s own state (settings, logs, certificates, installed engines) but leaves model files untouched.

Roadmap (high‑level upcoming work)

  • Full Windows ARM64 support, DGX Station compatibility.
  • New engine integrations: vLLM, EXO, ComfyUI, Unsloth, Tailscale.
  • Advanced routing features: cross‑engine OpenAI‑compatible routing, request queuing, QoS scheduler knobs, KV‑cache‑aware scheduling.
  • Usability: requester‑only clients, auto‑start on boot, more stability improvements.

License & contributions

  • Licensed under Apache 2.0.
  • Contributions are welcomed; see CONTRIBUTING.md, CODE_OF_CONDUCT.md, and the governance model for details.
  • Security: PAIR includes LAN discovery, PIN‑based trust bootstrap, and local HTTP endpoints. Review SECURITY.md before deploying on untrusted networks.

Bottom line – NVIDIA PAIR is a ready‑to‑use, cross‑platform router that lets you treat a handful of local machines as a single inference service, keeping all data on‑premises while handling engine installation, model management, and request scheduling for you.

Related

  • Project
    autonomous-ai/Personal-AI-Computer

    The repository provides open‑source hardware designs and assembly instructions for building high‑performance, locally‑run AI workstations (from a 2‑GPU desktop to an 8‑GPU rack). It pairs the hardware with open‑source inference software (Grid, vLLM, Ollama, llama.cpp) so users can run large language models on‑prem, avoiding cloud costs and preserving data privacy.

  • Project
    ollaya-dev/ollaya

    ollaya is a Rust‑based tool that serves non‑generative decision models locally, offering an Ollama‑like CLI, TypeSafe‑compatible HTTP API, and support for ONNX and GGUF back‑ends. It ships tiny ONNX graphs, pulls weights from Hugging Face, and provides calibrated, millisecond‑level predictions for structured questions, targeting AI agents and decision‑making workloads.

  • Dispatch
    vLLM Semantic Router Fusion primitive enables programmable multi‑model serving

    vLLM introduced the Fusion primitive for its Semantic Router, enabling programmable multi‑model panels, judging, and synthesis as a first‑class serving pattern.

  • Dispatch
    Mesh LLM: Peer‑to‑Peer Distributed LLM Inference on iroh

    Mesh LLM lets you pool GPUs across laptops, workstations and servers into a single OpenAI‑compatible inference API using iroh’s authenticated QUIC mesh, eliminating reliance on centralized AI providers.

  • Project
    Osmantic/ODS

    A local AI server stack that provides a one-command installation of inference engines, chat interfaces, and automation tools for private, self-hosted AI.