abetlen/llama-cpp-python

Python bindings for llama.cpp

What it solves

It provides a way to use the llama.cpp library within Python, allowing developers to run large language models (LLMs) locally with high performance and minimal overhead. It bridges the gap between the low-level C++ implementation of llama.cpp and the high-level flexibility of Python.

How it works

The project creates Python bindings for llama.cpp using a ctypes interface for low-level access. It offers a high-level Llama class that manages model loading and text generation. It supports various hardware acceleration backends (such as CUDA, Metal, Vulkan, and ROCm) to speed up inference and can download models in GGUF format directly from the Hugging Face Hub.

Who it’s for

Python developers who want to integrate local LLM inference into their applications, as well as those looking for an OpenAI-compatible local server for text, chat, and vision tasks.

Highlights

  • OpenAI Compatibility: Includes a high-level API and a web server that mimics the OpenAI API, making it a drop-in replacement for many existing tools.
  • Broad Hardware Support: Supports a wide range of acceleration backends including NVIDIA (CUDA), Apple Silicon (Metal), AMD (ROCm), and Vulkan.
  • Structured Output: Supports JSON mode and JSON Schema mode to constrain model responses to specific formats.
  • Multi-modal Capabilities: Supports vision-language models (like Llava and Moondream) for processing both text and images.
  • Ecosystem Integration: Compatible with popular frameworks like LangChain and LlamaIndex.

Related

  • Project
    go-skynet/go-llama.cpp

    Go bindings for llama.cpp that enable high-performance local LLM inference within Go applications using the GGUF format.

  • Dispatch
    llama.cpp and llama.app: Local LLM Inference and Ecosystem

    llama.cpp and its official home llama.app provide high-performance, hardware-agnostic local LLM inference, now featuring a simplified 'llama serve' command and integration with local coding agents like Pi.

  • Project
    ggml-org/llama.cpp

    A high-performance C/C++ implementation for local LLM and VLM inference, supporting a wide range of hardware through quantization and hardware acceleration.

  • Project
    withcatai/node-llama-cpp

    A Node.js library that provides pre-built bindings for llama.cpp, enabling developers to run LLMs locally with automatic hardware acceleration and TypeScript support.

  • Project
    ngxson/wllama

    A WebAssembly binding for llama.cpp that enables running LLMs, multimodal models, and tool calling directly in the web browser using WebGPU and Wasm.