lightseekorg/tokenspeed

TokenSpeed is a speed-of-light LLM inference engine.

What it solves

TokenSpeed is designed to provide high-performance LLM inference specifically optimized for agentic workloads. It aims to combine the raw speed of TensorRT-LLM with the ease of use found in vLLM, addressing the need for a production-ready engine that can handle the specific demands of AI agents.

How it works

TokenSpeed uses a unique architectural split between its control and execution planes:

  • Control Plane: Implemented in C++ as a finite-state machine. It manages the request lifecycle and KV cache ownership, using the type system to ensure safe resource management at compile time.
  • Execution Plane: Implemented in Python to allow researchers and engineers to iterate quickly.
  • Kernel System: A modular, pluggable system with a portable public API and centralized registry, including highly optimized implementations of Multi-head Latent Attention (MLA) for Blackwell GPUs.
  • Entrypoint: Uses an SMG-integrated AsyncLLM to handle CPU-side requests with low overhead.

Who it’s for

It is built for engineers and researchers deploying production-grade agentic workloads who require maximum inference throughput and low latency across diverse hardware, including NVIDIA and AMD GPUs.

Highlights

  • Separated Architecture: First engine to decouple the C++ control plane from the Python execution plane for both safety and velocity.
  • Modular Kernels: Treats kernels as first-class subsystems with an extensible plugin mechanism for different accelerators.
  • Hardware Support: Backed by the non-profit LightSeek Foundation with engineering support from NVIDIA and AMD.
  • High Performance: Capable of reaching high throughput (e.g., 580 TPS on Qwen3.5-397B-A17B) for agentic tasks.

Related

  • Project
    ModelTC/LightLLM

    LightLLM is a Python‑based, high‑performance inference and serving framework for large language models. It combines fast kernels (FlashAttention, Triton) with lightweight, token‑level KV‑cache management, supports multi‑GPU scaling, and adds research‑grade features like constrained decoding and SLA‑aware request scheduling.

  • Project
    flashinfer-ai/flashinfer

    FlashInfer is a GPU‑kernel library (with a Python package) that provides ultra‑fast, low‑precision attention, GEMM, MoE, and sampling operations for LLM inference. It automatically selects the best backend (FlashAttention‑2/3, cuDNN, CUTLASS, TensorRT‑LLM) for a wide range of NVIDIA GPUs, from Turing up to Blackwell, and is already used by popular inference servers like vLLM and TensorRT‑LLM.

  • Project
    NVIDIA/TensorRT-LLM

    TensorRT‑LLM is NVIDIA’s open‑source library that compiles large language (and visual‑generation) models into ultra‑fast TensorRT inference engines. It provides custom CUDA kernels, quantization, sparse attention, speculative decoding, and multi‑GPU parallelism, delivering 2‑10× higher token throughput while keeping accuracy. The Python‑friendly runtime integrates with Triton, Kubernetes, and Jetson, and includes docs and examples for models like GPT‑OSS‑120B, Llama‑4, DeepSeek‑R1, and diffusion models.

  • Project
    EricLBuehler/candle-vllm

    A high-performance inference and serving platform for local LLMs featuring an OpenAI-compatible API, multi-GPU/multi-node support, and aggressive KV cache compression.

  • Dispatch
    AI & Frontier Tech Roundup – Agentic Models, Robotics, and Emerging Inference Techniques

    Recent posts highlight a surge in agentic AI models, new robotics deployments, and advances in inference efficiency shaping the frontier of AI development.