chiennv2000/orthrus
Fast, lossless LLM inference via dual-view diffusion decoding.
What it solves
Orthrus addresses the sequential bottleneck of standard autoregressive Large Language Models (LLMs), where tokens are generated one by one. It aims to provide the high-speed parallel token generation typical of diffusion models without sacrificing the accuracy or predictive distribution of the original base model.
How it works
Orthrus uses a dual-architecture framework that combines an autoregressive view and a diffusion view. These two views share the exact same Key-Value (KV) cache, which eliminates redundant memory overhead. To ensure the output is identical to the base model, it employs an intra-model consensus mechanism. The parallel generation capability is added by fine-tuning only 16% of the model's parameters while keeping the base LLM frozen.
Who it’s for
Developers and researchers looking to accelerate LLM inference speed (throughput) without losing any generation quality, particularly for long-context tasks where traditional speculative decoding methods may struggle.
Highlights
- Lossless Generation: Guarantees that outputs match the base model's exact predictive distribution.
- Significant Speedup: Delivers up to 7.8× speedup over standard decoding.
- Memory Efficient: Uses a shared KV cache to maintain $O(1)$ memory cache overhead.
- Parameter Efficient: Only 16% of parameters are fine-tuned to enable parallel generation.