The Inference Frontier: 10× Faster Models to Self‑Optimizing AI — Baseten Talk Summary
Handling a 200 K‑Token Prompt
When a 200 000‑token request arrives, the system first checks whether the prompt (or part of it) has been seen before to reuse cached KV cache; if not, it routes the request to a prefill worker that creates the KV cache and returns the first token, then passes the cache to a separate set of GPUs for decode.
Cache‑Aware Routing and Disaggregated Prefill/Decode
Cache‑aware routing selects an instance with available prefill workers and ideally some cached input so that prefill can be skipped on at least part of the tokens. Disaggregating prefill and decode means one set of GPUs solely processes the input and builds the KV cache, while a separate set runs decode to generate tokens iteratively.
Speculative Decoding
A small speculator model runs a few ultra‑aggressive forward passes to predict several tokens; the main model then verifies those predictions in a single forward pass. Accepted speculator tokens are streamed to the user, reducing decode latency. The speculator is traffic‑specific; for coding workloads it achieves a high draft‑token acceptance rate.
Quantization and Error Cancellation
Quantization is lossy but speed improves. The main lossy optimization is quantization; preserving fidelity requires choosing which layers to quantize and calibrating to keep outliers. Quantization errors can cancel each other out: quantizing certain layers may produce errors that offset errors in other layers, yielding a model with better fidelity than quantizing fewer layers. This enables 20 % more throughput at NVFP4 while maintaining or improving quality relative to other quantized versions.
Achieving Up to 10× Faster Inference
Starting from a baseline of roughly 30–40 tokens per second for a trillion‑parameter model without optimizations, stacking techniques such as quantization (≈30‑40 % gain per step), a 2× speculator, disaggregated prefill/decode (≈2×), and better kernels/runtime can yield 8× or more. With sufficient traffic and hardware, gains of 20 %, 100 %, or even 200 % over a naive deployment are possible, and end‑to‑end speedups of up to 10× are achievable.
NVIDIA Dynamo and KV‑Aware Routing
Dynamo is an open‑source NVIDIA library that moves KV cache around a cluster; it is a developer toolkit, not an out‑of‑the‑box optimizer. KV‑aware routing uses Dynamo to place KV cache where it is needed, supporting disaggregation and reducing latency caused by cache transfers between nodes.
Model Parallelism, Auto‑Tuning, and Mega Kernels
Tensor parallelism shards a model across GPUs requiring high‑bandwidth interconnects; expert parallelism places whole experts on each GPU with a replicated router, reducing inter‑GPU communication. Pipeline parallelism is used only when a model exceeds single‑node memory, forcing multi‑node splits. Auto‑tuning sweeps thread counts, shared memory, and parallelism configurations to find the best latency/throughput trade‑off empirically. Mega kernels (fusing many operations) are difficult to write and often slower than modular kernels because launch overhead and optimization losses outweigh benefits.
Hardware Trends: Rubin, GPUs vs. ASICs
Future GPUs such as Rubin emphasize faster CPU‑GPU and GPU‑GPU interconnects, making KV cache offloading and systems‑level optimizations more important. While GPUs are gaining ASIC‑like tensor cores and specialized instructions, they remain programmable; fully burning model weights into silicon would prevent fine‑tuning, quantization, and new checkpoints, so a spectrum of specialization is more realistic than a pure ASIC shift.
Video Generation Challenges
Generating coherent long‑form video faces a quadratic attention bottleneck: at 16 fps and 480p, even five seconds of video involves ~35 000 tokens, leading to attention over millions of tokens for longer clips. Full attention over such sequences is infeasible, pushing research toward either sparse/spatial‑temporal attention or autoregressive video models. Current open‑source autoregressive video models suffer from quality drift, while diffusion models remain limited in scale compared to closed‑source alternatives like Veo or Kling.
Continual Learning and Self‑Optimizing Models
Continual learning can be approached by updating model weights or by KV‑cache compaction; the latter preserves knowledge without weight changes, enabling inference‑time learning. GLM‑5.2 demonstrated self‑optimization: it was used to profile its own inference, identify bottleneck kernels, write new kernels, and repeat the cycle, with some GPU kernels in Baseten’s inference engine authored by GLM‑5.2 itself. This illustrates a loop where models help optimize the infrastructure that runs them.
Closing Thoughts
Inference engineering is evolving from simply making a model run fast to a systems problem that spans kernels, hardware interconnects, model design, and continual learning. As models grow and hardware advances, the race to squeeze another 10×, 100×, or 1000× out of inference will continue, driven by faster networking, better KV‑cache management, and models that can improve their own serving stack.