hybridgroup/yzma

Go with your own intelligence - Go applications that directly integrate llama.cpp for local inference using hardware acceleration.

What it solves

yzma provides a way for Go developers to integrate local AI model inference directly into their applications without needing a C compiler (CGo) or managing external model servers. It simplifies the process of running Large Language Models (LLMs) and Vision Language Models (VLMs) locally on various hardware.

How it works

The project acts as a Go wrapper around llama.cpp, utilizing purego and ffi to call the llama.cpp libraries in the same process. This allows the application to leverage hardware acceleration (such as CUDA, Metal, or Vulkan) for high performance while maintaining a standard Go build process.

Who it’s for

Go developers who want to build applications with embedded local AI capabilities, avoiding the complexity of containers or external inference servers.

Highlights

  • CGo-free integration: Uses purego and ffi so no C compiler is needed for building Go programs.
  • Broad hardware support: Supports CUDA, Metal, Vulkan, HIP, ROCm, SYCL, and OpenCL across Linux, macOS, and Windows.
  • Local inference: Runs models in the same process as the Go application for maximum performance.
  • Multimodal capabilities: Supports both text-based LLMs and Vision Language Models (VLMs) using GGUF format models.
  • Simplified model management: Includes a command-line tool to download GGUF models directly from Hugging Face.

Related

  • Project
    go-skynet/go-llama.cpp

    Go bindings for llama.cpp that enable high-performance local LLM inference within Go applications using the GGUF format.

  • Project
    abetlen/llama-cpp-python

    Python bindings for llama.cpp that enable local LLM inference with an OpenAI-compatible API and support for multiple hardware acceleration backends.

  • Project
    SciSharp/LLamaSharp

    A cross-platform C# library based on llama.cpp that allows developers to run LLaMA and other LLMs locally on CPU and GPU with high-level APIs.

  • Dispatch
    llama.cpp and llama.app: Local LLM Inference and Ecosystem

    llama.cpp and its official home llama.app provide high-performance, hardware-agnostic local LLM inference, now featuring a simplified 'llama serve' command and integration with local coding agents like Pi.

  • Project
    withcatai/node-llama-cpp

    A Node.js library that provides pre-built bindings for llama.cpp, enabling developers to run LLMs locally with automatic hardware acceleration and TypeScript support.