Andyyyy64/whichllm

whichllm – Pick the best LLM that actually runs on your hardware

What it does – whichllm is a command‑line tool that inspects the CPU, GPU and RAM of your machine (or a GPU you pretend to have) and returns a ranked list of Hugging Face models that will fit and run well locally. The ranking is based on real benchmark scores, model size, quantisation, estimated generation speed and how comfortably the model fits in memory.

How it works –

1. Detects hardware via NVIDIA/AMD/Apple‑Silicon libraries and OS queries.
2. Queries the Hugging Face API for recent models, filters by format (GGUF, AWQ, GPTQ, FP16/BF16, etc.).
3. Merges benchmark data from several public leader‑boards (LiveBench, Aider, Chatbot Arena, Open LLM Leaderboard) and applies confidence‑based discounts for self‑reported or inherited scores.
4. Estimates VRAM usage (weights + KV‑cache + activations + framework overhead) and token‑per‑second speed based on bandwidth, quantisation and whether the model runs fully on‑GPU, with partial off‑load or CPU‑only.
5. Scores each candidate (0‑100) and outputs a table, JSON, or Markdown.

Key features –

• Auto‑detect or simulate any GPU (e.g. --gpu "RTX 4090").
• Filter for full‑GPU fits (--gpu-only), speed thresholds (--speed usable), or specific quantisations.
• One‑command chat (whichllm run …) and ready‑to‑paste Python snippets (whichllm snippet …).
• Planning helpers: whichllm plan "llama 3 70b" tells you the GPU you need; whichllm upgrade … compares hardware upgrades.
• Output formats: plain table, GitHub‑flavoured Markdown, or machine‑readable JSON for pipelines.
• Live data with a cache (6 h for model list, 24 h for benchmarks).

Typical workflow –

# Quick recommendation for the current machine
whichllm

# Pretend you have a new GPU before buying
whichllm --gpu "RTX 4090"

# Get a ready‑to‑run chat with the best model for your rig
whichllm run

# Export the top result as JSON for a script
whichllm --top 1 --json

Why it matters – Many tools simply pick the largest model that fits in VRAM. whichllm goes further: it evaluates quality (benchmark scores), recency (newer models are favoured), and practical speed to suggest the model that will give the best experience on your exact hardware.

Installation – Available via uv, brew, or pip (e.g. uvx whichllm@latest).

---

All details are taken from the project's README; no additional claims are added.