IIIIIllllIIIIIlllll/llama.cpp-hub

An extension utility for llama.cpp, used with 3090*2 + Strix Halo. llama.cpp的拓展小工具，自用于3090*2 + Strix Halo。

What it solves

llama.cpp-hub is a graphical web interface for llama.cpp, designed to simplify the management and operation of GGUF models. It removes the need to interact with complex command-line arguments and remote server terminal operations, providing a centralized hub for managing multiple models and multiple servers.

How it works

The project acts as a wrapper around llama.cpp sub-processes. It uses a Java 21 and Netty 4.1 backend to manage independent inference processes for each loaded model. It exposes a unified API gateway (port 8080) that routes requests to the various model processes, providing compatibility with OpenAI and Anthropic API formats.

Who it’s for

  • Users who find remote server terminal operations cumbersome.
  • People managing multiple machines running llama.cpp across a network.
  • Users who prefer to compile their own versions of llama.cpp but want to avoid organizational chaos.
  • Those who struggle to remember the numerous configuration parameters of llama.cpp.

Highlights

  • Multi-Model Management: Load and unload GGUF models with saved startup configurations and sampling presets.
  • Unified API Gateway: A single backend exposing compatible APIs for OpenAI and Anthropic, allowing existing SDKs to be used with a simple address change.
  • Remote Node Aggregation: Ability to aggregate multiple llama.cpp-hub instances across different servers into a single management entry point.
  • Web Management Panel: Real-time status, WebSocket logs, token consumption statistics, and performance benchmarking tools.
  • PWA Support: Installable as a Progressive Web App for a native-like desktop experience.
  • Download Manager: Built-in support for HTTP breakpoint resume for transferring GGUF models.
  • MCP Support: Built-in Model Context Protocol (MCP) server for testing tool calling capabilities.

Related

  • Dispatch
    llama.cpp Model Management and Router Mode

    llama.cpp server introduces router mode, enabling dynamic loading, unloading, and switching between multiple GGUF models without restarting the server.

  • Project
    ngxson/wllama

    A WebAssembly binding for llama.cpp that enables running LLMs, multimodal models, and tool calling directly in the web browser using WebGPU and Wasm.

  • Dispatch
    llama.cpp and llama.app: Local LLM Inference and Ecosystem

    llama.cpp and its official home llama.app provide high-performance, hardware-agnostic local LLM inference, now featuring a simplified 'llama serve' command and integration with local coding agents like Pi.

  • Project
    hybridgroup/yzma

    A Go library that integrates llama.cpp for fully local LLM and VLM inference with hardware acceleration, requiring no CGo or external servers.

  • Project
    Vali-98/ChatterUI

    A native mobile frontend for LLMs that allows users to run models on-device using llama.cpp or connect to various commercial and open-source APIs.