Luce-Org/lucebox

LLM speculative inference server for heterogeneous hardware & consumer GPUs

Lucebox – High‑performance LLM inference for consumer GPUs and heterogeneous rigs

What it is Lucebox is an open‑source inference engine written in modern C++ (C++17) that targets both NVIDIA (CUDA) and AMD (HIP) GPUs. It focuses on speculative inference: custom kernels that pre‑fill tokens faster and decode more efficiently, often by using a lightweight “drafter” model to guess the next token before the full model finishes its work. The project supplies a suite of optimizations (DFlash2, DSpark, PFlash, KVFlash, Megakernel, etc.) and a set of ready‑made model‑hardware profiles.

Key features

  • Speculative prefill & decoding – a drafter runs ahead of the main model, cutting prefill latency by up to 6× and boosting decode speed by several times.
  • Hardware‑specific kernels – tuned for a wide range of GPUs/APUs, from consumer Radeon R9700 and RTX 3090/4090/5090 to DGX‑Spark and Jetson AGX Thor.
  • Heterogeneous execution – can split work across multiple devices (e.g., an AMD GPU for the drafter and another for the main model) and even use SSD‑resident “experts” for very long contexts.
  • Paged attention & continuous batching – keeps many client requests alive with low overhead, useful for serving chat‑style workloads.
  • Vision support – image‑question pipelines are integrated, allowing multimodal inference on the same engine.
  • Model‑agnostic – works with any GGUF‑compatible checkpoint; the README lists dozens of tested models (Qwen 3.8 27B, DeepSeek V4, Laguna XS 33B, Gemma 4 26/31B, etc.) and the corresponding drafter/quantization.

Performance highlights (from the README)

Hardware Model Speed
Radeon R9700 Qwen 3.8 27B (DFlash2) 208 tok/s (HumanEval)
RTX 5090 Qwen 3.8 27B 110 tok/s for a 26 k‑token prompt
2× RTX 3090 (NVLink) Qwen 3.8 + DFlash2 79.7 tok/s, 2.16× faster decode
Mixed AMD (R9700 + Strix Halo) DeepSeek V4 86 tok/s decode, 788 tok/s prefill
Vision (R9700 + Strix Halo) Qwen 3.8 27B vision 3.2× throughput vs a DGX Spark

How to get started

  1. Install – the repo provides CMake build files that automatically select CUDA or HIP based on your system.
  2. Pick a profile – the server/docs/RECOMMENDED_SETUPS.md file lists ready‑made command‑line flags for each GPU/model combination (e.g., --profile ds41-lucebox).
  3. Run the server – a lightweight HTTP server ships with the code; you can send standard OpenAI‑compatible chat/completion requests.
  4. Extend – the modular kernel design lets you add new speculative drafters or integrate other quantization formats.

Who might use it

  • Researchers or hobbyists who want the fastest possible inference on a single consumer GPU.
  • Developers building private LLM services that need low latency for many concurrent users.
  • Anyone interested in experimenting with speculative decoding, mixed‑device pipelines, or multimodal (image‑text) inference without buying a data‑center‑grade server.

Where to find more

  • Website & blog – https://www.lucebox.com/ (detailed benchmark write‑ups)
  • Model weights – hosted on Hugging Face under the Lucebox organization.
  • Community – Discord link in the badge bar for real‑time help and performance tips.

All information above is taken directly from the repository’s README; no external assumptions have been added.

Related

  • Project
    kvcache-ai/ktransformers

    KTransformers is a research framework for efficient LLM inference and fine-tuning via CPU-GPU heterogeneous computing, enabling large MoE models like DeepSeek-R1 to run on a single consumer GPU (24GB VRAM) with 3-28x speedups, and offering fine-tuning with 6-12x speedup over ZeRO-Offload.

  • Project
    guqiong96/Lvllm

    An integration of vLLM and lk_moe that enables large MoE models to run across CPU and GPU memory using NUMA-aware scheduling.

  • Project
    turboderp-org/exllamav3

    ExLlamaV3 is a Python library for fast, low‑memory inference of large language models on consumer GPUs. It provides QTIP‑based EXL3 quantization (2‑8 bit), parallel and CPU‑offloaded execution, dynamic batching, and multimodal support, with integrations for Hugging Face, Transformers, and an OpenAI‑compatible API via TabbyAPI.

  • Project
    EricLBuehler/candle-vllm

    A high-performance inference and serving platform for local LLMs featuring an OpenAI-compatible API, multi-GPU/multi-node support, and aggressive KV cache compression.

  • Project
    alibaba/rtp-llm

    A high-performance LLM inference acceleration engine developed by Alibaba that optimizes throughput and latency for production-scale deployments.