hybridgroup/yzma

Go with your own intelligence - Write Go applications that directly integrate llama.cpp for local inference using hardware acceleration on Linux, macOS, Windows, & WebAssembly.

What it solves

yzma provides a way for Go developers to integrate local LLM and Vision Language Model (VLM) inference directly into their applications without needing a C compiler (CGo) or external model servers. It simplifies the process of running high-performance, hardware-accelerated AI models on a wide variety of platforms, including desktop OSs and web browsers.

How it works

The project acts as a Go wrapper for llama.cpp, utilizing the purego and ffi packages to call llama.cpp libraries in the same process. This allows the application to leverage hardware acceleration (such as CUDA, Metal, Vulkan, and OpenVINO) without requiring CGo. It also supports WebAssembly (WASM) via TinyGo, enabling models to run entirely in the browser using the CPU or WebGPU.

Who it’s for

Go developers who want to build applications with embedded local AI capabilities (text and vision) across multiple platforms (Linux, macOS, Windows, and Browser) without the complexity of managing external dependencies or C-based build chains.

Highlights

  • CGo-free integration: Uses purego and ffi to avoid the need for a C compiler during build.
  • Broad hardware acceleration: Supports CUDA, Metal, OpenVINO, Vulkan, HIP, ROCm, and SYCL.
  • Multimodal capabilities: Supports both text-based LLMs and Vision Language Models (VLMs).
  • Browser support: Runs in the browser via WebAssembly and WebGPU using TinyGo.
  • Direct process execution: Eliminates the need for external servers for lower latency and higher performance.
  • C-library management: Includes a CLI tool to download and verify prebuilt llama.cpp libraries.

Related

  • Project
    go-skynet/go-llama.cpp

    Go bindings for llama.cpp that enable high-performance local LLM inference within Go applications using the GGUF format.

  • Project
    abetlen/llama-cpp-python

    Python bindings for llama.cpp that enable local LLM inference with an OpenAI-compatible API and support for multiple hardware acceleration backends.

  • Project
    ngxson/wllama

    A WebAssembly binding for llama.cpp that enables running LLMs, multimodal models, and tool calling directly in the browser via WebGPU or Wasm SIMD.

  • Project
    withcatai/node-llama-cpp

    A Node.js library that provides easy-to-use bindings for llama.cpp, enabling developers to run LLMs locally with automatic hardware acceleration.

  • Dispatch
    llama.cpp and llama.app: Local LLM Inference and Ecosystem

    llama.cpp and its official home llama.app provide high-performance, hardware-agnostic local LLM inference, now featuring a simplified 'llama serve' command and integration with local coding agents like Pi.