lightseekorg/tokenspeed
TokenSpeed is a speed-of-light LLM inference engine.
What it solves
TokenSpeed is designed to solve the performance bottlenecks of LLM inference, specifically for agentic workloads. It aims to combine the high performance of TensorRT-LLM with the ease of use found in vLLM.
How it works
TokenSpeed employs several core components to optimize inference:
- Modeling layer: Uses a local-SPMD design and a static compiler that automatically generates collective communication, removing the need for users to manually write parallelism logic.
- Scheduler: Combines a C++ control plane with a Python execution plane, managing the request lifecycle and KV cache ownership through a finite-state machine.
- Kernels: Features a pluggable, layered kernel system with a centralized registry, including a high-performance implementation of Multi-head Latent Attention (MLA) for Blackwell GPUs.
- Entrypoint: Utilizes an SMG-integrated AsyncLLM for low-overhead CPU-side request handling.
Who it’s for
It is intended for developers and organizations deploying production-grade agentic workloads that require extreme inference speed and high throughput.
Highlights
- High performance on agentic workloads, achieving up to 580 TPS on Qwen3.5-397B-A17B.
- Support for FP4 inference on NVIDIA and AMD hardware.
- Automatic parallelism logic generation via a static compiler.
- Optimized KV cache resource reuse enforced at compile time.