AstraNetLab/CacheRoute

What it solves

CacheRoute reduces the heavy prefill overhead and redundant computation that occur when multiple LLM requests repeatedly use the same long external knowledge texts. Instead of recomputing these prompts every time, it enables the reuse of Key-Value (KV) caches across different LLM systems to lower latency and increase throughput.

How it works

Built on vLLM and LMCache, CacheRoute implements a Knowledge Delivery Network (KDN) architecture. It uses dedicated KDN servers to store and manage KV cache blocks for popular knowledge.

When a request arrives, the system follows this workflow:

1. Scheduler: Analyzes the knowledge requirement and routes the request to the most efficient resource pool.
2. Proxy: Predicts the cost of text-based vs. KV cache-based injection and dynamically chooses the best strategy based on real-time compute and network load.
3. KDN Server: Injects the pre-computed KV cache blocks if the reuse strategy is selected.
4. Instance: Forwards the request to the vLLM and LMCache backend for final execution.

Who it’s for

It is designed for knowledge-intensive LLM services, such as browser-based AI assistants and knowledge-based QA systems, where the same reference materials are frequently accessed by many users.

Highlights

• Compute-Network Coordination: Dynamically switches between recomputation and cache injection based on current system load.
• Cross-System Reuse: Shares reusable knowledge across different LLM systems via KDN servers.
• Knowledge-Oriented Routing: Routes requests based on knowledge availability and system topology.
• Observability Tools: Includes a Proxy browser UI and an Instance resource dashboard for monitoring control-plane state and resource snapshots.