ggml-org/llama.cpp

LLM inference in C/C++

llama.cpp – Fast LLM inference in pure C/C++

What it is – llama.cpp is a library and set of command‑line tools that run large language models (LLMs) locally. It is written in plain C/C++ with no external runtime dependencies, and it targets a huge variety of CPUs and GPUs (Apple Metal, x86 AVX/AVX2/AVX‑512/AMX, RISC‑V extensions, CUDA, Vulkan, SYCL, etc.). The goal is to provide state‑of‑the‑art inference performance while keeping the setup as simple as possible.

Key features

  • Supports the GGML/​GGUF model formats and can quantize models to 1.5‑8 bit integers for lower memory and faster inference.
  • Runs on everything from laptops (Apple Silicon, x86, RISC‑V) to cloud GPUs (NVIDIA, AMD, Intel, Moore Threads, Ascend) and even in the browser via WebGPU.
  • Provides a tiny CLI (llama-cli) for direct prompting, a REST‑compatible server (llama‑server) that mimics the OpenAI API, and a C library (libllama) for embedding the engine in other programs.
  • Extensive model support: LLaMA 1‑3, LLaMA 2, Mistral, Mixtral, Falcon, Gemma, Yi, Qwen, many multilingual and code models, plus a growing list of multimodal vision‑language models (LLaVA, BakLLaVA, Moondream, etc.).
  • A rich ecosystem of language bindings (Python, Go, Node, Rust, .NET, Java, Swift, Ruby, etc.) and UI front‑ends (LMStudio, Ollama, Jan, KoboldCPP, LocalAI, etc.).

Getting started

  1. Install – Use a package manager (brew, nix, winget, conda‑forge), pull a pre‑built binary from the releases page, run the provided Docker image, or build from source (see docs/install.md and docs/build.md).
  2. Obtain a model – Download a GGUF model from Hugging Face (llama-cli -hf <repo>/<model>-GGUF) or convert your own PyTorch checkpoint with tools like ggify.
  3. Run a prompt
    llama-cli -m my_model.gguf "Explain quantum computing in one sentence"
  4. Run a server (compatible with OpenAI‑style clients):
    llama-server -hf ggml-org/gemma-3-1b-it-GGUF
    The server can be front‑ended by any OpenAI client library.

Why it matters

  • Enables private, offline LLM usage without heavyweight frameworks.
  • Lowers the hardware barrier: even a modest CPU can run 4‑7 B‑parameter models thanks to aggressive quantization.
  • Serves as the reference implementation for the GGML ecosystem, driving many downstream projects (bindings, UI apps, cloud‑native LLMOps platforms, etc.).

Resources

  • Docs – installation, build options, backend specifics, multimodal support, model‑adding guide.
  • Community – discussions for new features, packaging, WebUI, and collaborations (e.g., NVIDIA MXFP4 support).
  • Bindings & UI list – a curated set of language bindings and user interfaces that rely on llama.cpp.

All information above is taken directly from the repository’s README.