Luce-Org/lucebox

LLM speculative inference server for heterogeneous hardware & consumer GPUs

What it solves

Lucebox is a high-performance inference engine designed to bring fast, local LLM execution to heterogeneous consumer hardware, including NVIDIA GPUs, AMD GPUs, and APUs (like Strix Halo). It addresses the performance bottlenecks of running large models on consumer-grade equipment by using speculative inference and custom hardware-tuned kernels.

How it works

The engine employs several advanced optimization techniques to maximize throughput and reduce latency:

  • Speculative Inference: Uses smaller "drafter" models to predict tokens, which are then verified by the larger target model, significantly speeding up decoding.
  • Custom Kernels: Implements specialized kernels (e.g., DFlash, PFlash, KVFlash, Megakernel) tuned for specific hardware architectures (CUDA and HIP).
  • Memory Management: Utilizes paged attention and continuous batching to handle multiple concurrent requests efficiently.
  • Heterogeneous Execution: Supports distributing model execution across different types of hardware (e.g., combining an AMD GPU and an APU).

Who it’s for

It is intended for developers and AI enthusiasts who want to run powerful LLMs locally on consumer hardware with high token-per-second rates, as well as those integrating LLMs into coding clients via its OpenAI-compatible API.

Highlights

  • Broad Hardware Support: Compatible with NVIDIA (Ampere, Ada, Blackwell) and AMD (RDNA3, RDNA4) architectures.
  • High Throughput: Achieves significant speedups (e.g., over 200 tok/s on certain AMD hardware).
  • OpenAI-Compatible API: Allows easy integration with various coding clients and harnesses.
  • Speculative Prefill and Decoding: Optimizes both the initial prompt processing and the token generation phase.
  • ** uma-GPU and Mixed-GPU Profiles**: Provides pre-configured setups for various hardware combinations.

Related

  • Project
    kvcache-ai/ktransformers

    A framework for efficient LLM inference and fine-tuning that uses CPU-GPU heterogeneous computing to run ultra-large MoE models on consumer hardware.

  • Project
    turboderp-org/exllamav3

    An inference library for running local LLMs on consumer GPUs featuring EXL3 quantization, parallel inference, and CPU offloading for large MoE models.

  • Project
    ztxz16/fastllm

    A high-performance C++ LLM inference engine that enables hybrid deployment of dense and MoE models across GPU, CPU, NUMA, and disk to maximize hardware utilization.

  • Project
    flashinfer-ai/flashinfer

    A high-performance GPU kernel library and generator for LLM inference that optimizes attention, GEMM, and MoE operations across diverse NVIDIA GPU architectures.

  • Project
    EricLBuehler/candle-vllm

    A high-performance inference and serving platform for local LLMs featuring an OpenAI-compatible API, multi-GPU/multi-node support, and aggressive KV cache compression.