ggml-org/ggml

Tensor library for machine learning

What it solves

It provides a simple, portable, and efficient way to handle tensors for machine learning, removing the need for complex setups or heavy dependencies when implementing ML models.

How it works

Written in plain C/C++, the library implements SIMD-optimized kernels for various architectures (x86, ARM, RISC-V) and supports a wide range of backends including CPUs, GPUs, NPUs, and web browsers. It utilizes 2- to 8-bit integer quantization and microscaling formats (MXFP4 and NVFP4) to reduce model size and memory usage, while ensuring zero memory allocations during runtime for maximum efficiency.

Who it’s for

Developers who need a lightweight, cross-platform tensor library for machine learning that can run on everything from high-end hardware to WebAssembly.

Highlights

  • Broad cross-platform support (x86, ARM, RISC-V, LoongArch, PowerPC, s390x, and WebAssembly).
  • Zero memory allocations during runtime.
  • Support for 2- to 8-bit integer quantization.
  • No external dependencies.

Related

  • Dispatch
    Introduction to ggml

    ggml is a lightweight, C/C++ machine learning library optimized for Transformer inference and on-device LLM execution across diverse hardware backends.

  • Project
    monatis/clip.cpp

    A dependency-free C/C++ implementation of CLIP for efficient inference on memory-constrained devices and serverless deployments.

  • Project
    foldl/chatllm.cpp

    A C++ LLM inference engine based on ggml that enables memory-efficient local execution of multimodal models with RAG and quantization support.

  • Project
    zml/zml

    A production inference stack that decouples AI workloads from proprietary hardware, allowing a single codebase to run models on NVIDIA, AMD, Intel, and TPU/Trainium accelerators.

  • Project
    amd/RyzenAI-SW

    A suite of tools and runtime libraries for optimizing and deploying AI inference on AMD Ryzen AI-based PCs to maximize hardware acceleration.