lightseekorg/tokenspeed

TokenSpeed is a speed-of-light LLM inference engine.

What it solves

TokenSpeed is a high-performance LLM inference engine specifically optimized for agentic workloads. It aims to provide the extreme performance of TensorRT-LLM while maintaining the ease of use and usability of vLLM.

How it works

TokenSpeed utilizes a unique architectural split between its control and execution planes:

  • Control Plane: Implemented in C++ as a finite-state machine, using the type system to enforce safe resource management (such as KV cache state and request lifecycles) at compile time.
  • Execution Plane: Implemented in Python, allowing for faster iteration and lower cognitive load for developers.
  • Modeling Layer: Uses a local-SPMD design with a static compiler that generates collective communication based on module-boundary placement annotations, removing the need for users to manually write parallelism logic.
  • Kernels: Features a pluggable, layered kernel system with a portable public API and a centralized registry, including highly optimized MLA (Multi-head Latent Attention) implementations for Blackwell GPUs.
  • Entrypoint: Uses an SMG-integrated AsyncLLM for low-overhead CPU-side request handling.

Who it’s for

Researchers and engineers building production-grade agentic AI systems that require extreme inference speed and efficient resource management.

Highlights

  • Separation of Planes: First engine to separate the C++ control plane from the Python execution plane for correctness and velocity.
  • High Performance: Achieves high throughput (e.g., 580 TPS on Qwen3.5-397B-A17B) and competes with TensorRT-LLM on Pareto curves for agentic workloads.
  • Automated Parallelism: Static compiler handles collective communication, simplifying the model deployment process.
  • Modular Kernels: Pluggable kernel system supporting heterogeneous accelerators.

Related

  • Project
    alibaba/rtp-llm

    A high-performance LLM inference acceleration engine developed by Alibaba that optimizes throughput and latency for production-scale deployments.

  • Project
    NVIDIA/TensorRT-LLM

    An optimization library for LLMs and visual generation models that uses specialized kernels and an efficient runtime to maximize inference throughput and minimize latency on NVIDIA GPUs.

  • Project
    UbiquitousLearning/mllm

    A fast, lightweight multimodal LLM inference engine for mobile and edge devices that optimizes vision-language model execution across various NPUs and GPUs.

  • Project
    EricLBuehler/candle-vllm

    A high-performance inference and serving platform for local LLMs featuring an OpenAI-compatible API, multi-GPU/multi-node support, and aggressive KV cache compression.

  • Project
    xLLM-AI/xllm

    xLLM is an open‑source C++ inference framework that delivers high‑throughput, low‑latency LLM serving on Chinese AI accelerators (Ascend, Cambricon, Moore Threads, etc.). It separates a brpc‑based service layer from a hardware‑optimized engine, includes hybrid KV‑cache management via Mooncake, and provides ready‑to‑run deployment scripts for models such as GLM‑5.3‑Flash and MiniMax‑M3. The project is Apache‑2.0 licensed and backed by JD.com and several Chinese university labs.