ggml-org/llama.cpp

What it solves

llama.cpp provides a way to run Large Language Models (LLMs) and Vision Language Models (VLMs) locally on a wide variety of hardware with minimal setup and high performance. It removes the need for heavy dependencies, allowing users to run powerful AI models on their own machines, including laptops and cloud servers.

How it works

Written in plain C/C++, the project is built on top of the ggml library. It uses advanced quantization (ranging from 1.5-bit to 8-bit) to reduce the memory footprint and speed up inference. To maximize performance, it includes specialized optimizations for various architectures, including Apple Silicon (Metal), NVIDIA GPUs (CUDA), AMD GPUs (HIP), and various CPU instructions (AVX, RVV).

Who it’s for

Developers and AI enthusiasts who want to run LLMs and VLMs locally without relying on cloud APIs, as well as those looking for optimize inference performance on consumer-grade hardware.

Highlights

• Wide Hardware Support: Supports a vast array of backends including CUDA, Metal, Vulkan, SYCL, and various CPUs (x86, RISC-V, Apple Silicon).
• Quantization: Offers multiple integer quantization levels (1.5-bit to 8-bit) to fit larger models into smaller memory.
• Dependency-Free: A plain C/C++ implementation with no external dependencies.
• OpenAI-Compatible API: Includes a server mode (llama serve) that provides an OpenAI-compatible REST API.
• Hybrid Inference: Supports CPU+GPU hybrid inference to run models that exceed the total VRAM capacity.