Introduction to ggml

ggml is a low-level machine learning (ML) library written in C and C++ designed specifically for Transformer inference. It serves as the underlying engine for popular on-device LLM projects including llama.cpp, whisper.cpp, ollama, Jan, LM Studio, and GPT4All.

Core Advantages of ggml

ggml is designed for minimalism and efficiency, making it ideal for deployment on resource-constrained hardware. Its primary benefits include:

  • Minimal Footprint: The core library is self-contained in fewer than five files, and the compiled binary size is under 1MB.
  • Simplified Compilation: It requires only GCC or Clang for basic compilation without GPU support, avoiding complex build tools.
  • Broad Hardware Compatibility: It supports x86_64, ARM, Apple Silicon, CUDA, and other hardware architectures.
  • Memory Efficiency: The library minimizes overhead for tensor storage and computation and supports quantized tensors to reduce memory usage and potentially improve performance.

Technical Trade-offs

As a low-level library in active development, ggml has specific limitations:

  • Backend Variance: Not all tensor operations are supported across all backends (e.g., an operation may work on CPU but not on CUDA).
  • Development Complexity: Using ggml requires deep knowledge of low-level programming.
  • Stability: Because it is under active development, breaking changes are expected.

Key Terminology and Concepts

Understanding ggml requires familiarity with several low-level abstractions that provide granular control over performance:

  • ggml_context: A container for tensors, graphs, and optional data.
  • ggml_cgraph: A computational graph representing the order of operations to be executed by a backend.
  • ggml_backend: An interface for executing computation graphs, with implementations for CPU (default), CUDA, Metal, Vulkan, and RPC.
  • ggml_backend_buffer_type: A memory allocator connected to a specific backend (e.g., allocating GPU memory).
  • ggml_backend_buffer: A buffer allocated by the buffer_type that can hold data for multiple tensors.
  • ggml_gallocr: A graph memory allocator used to efficiently manage tensors within a computation graph.
  • ggml_backend_sched: A scheduler that distributes computations across multiple backends (e.g., splitting work between CPU and GPU) and automatically assigns unsupported operations to the CPU.

Implementation Workflow

Basic Computation

For simple operations, such as matrix multiplication, the ggml workflow involves:

  1. Allocating a ggml_context for tensor data.
  2. Creating tensors and assigning data.
  3. Defining a ggml_cgraph for the operation (e.g., ggml_mul_mat).
  4. Executing the computation via ggml_graph_compute_with_ctx.
  5. Retrieving the results and freeing memory.

Backend-Driven Computation

When utilizing a specific hardware backend (like CUDA), the process is more detailed to ensure efficient device memory management:

  1. Initializing the ggml_backend.
  2. Allocating a ggml_context for tensor metadata only.
  3. Creating tensor metadata (shapes and types).
  4. Allocating a ggml_backend_buffer to store the tensors on the device.
  5. Copying tensor data from RAM to the backend buffer.
  6. Creating a ggml_cgraph and a ggml_gallocr for graph allocation.
  7. Executing the computation via ggml_backend_graph_compute.
  8. Copying the result tensor back from the device buffer to RAM.

Debugging the Computational Graph

Developers can inspect the ggml_cgraph to verify the order of operations. The library provides ggml_graph_print to output the graph to the console, and ggml_graph_dump_dot to export the graph in Graphviz dot format for visual rendering.

Sources