BlinkDL/RWKV-LM

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

What it solves

RWKV is designed to combine the performance of Transformer-based Large Language Models (LLMs) with the efficiency of Recurrent Neural Networks (RNNs). It eliminates the need for a KV-cache, which typically consumes significant VRAM and slows down inference as context grows, allowing for constant-space and linear-time complexity during generation.

How it works

RWKV is a 100% attention-free RNN architecture. It can be trained in parallel like a GPT-style Transformer (GPT mode) but operates as an RNN during inference (RNN mode), where only the hidden state at position $t$ is needed to compute the state at position $t+1$. The latest version, RWKV-7 "Goose", acts as a meta-in-context learner that updates its state via in-context gradient descent at every token.

Who it’s for

This project is for AI researchers and developers who need high-performance LLMs that are computationally efficient, particularly for those deploying on mobile or desktop devices where VRAM is limited, or for those requiring "infinite" context lengths.

Highlights

  • Linear-Time & Constant-Space: No KV-cache required, ensuring constant speed and VRAM usage regardless of sequence length.
  • Parallelizable Training: Can be trained like a Transformer for high throughput.
  • High Inference Speed: Capable of extremely high token-per-second rates (e.g., 10,000+ tokens/s for certain configurations on high-end GPUs).
  • LFAI Project: A Linux Foundation AI project, making it totally free and open.
  • Multimodal Support: Suitable for both text LLM and multimodal applications.

Related

  • Project
    BlinkDL/ChatRWKV

    A chat interface and inference implementation for the RWKV language model, an RNN-based alternative to Transformers that offers similar quality with higher speed and lower VRAM usage.

  • Dispatch
    RWKV Architecture Integration into Hugging Face Transformers

    Hugging Face announced the integration of the RWKV RNN‑Transformer hybrid model into the Transformers library, enabling open‑source, long‑context language models that combine RNN efficiency with transformer performance.

  • Project
    Zefan-Cai/R-KV

    R-KV is a redundancy-aware KV-cache compression method for reasoning models that reduces memory usage by up to 90% and boosts throughput without losing accuracy.

  • Project
    rednote-machine-learning/RedKnot

    A model-aware long-context execution framework that reduces LLM serving latency and compute costs through head-aware KV reuse and sparse execution.

  • Project
    NX-AI/xlstm

    xLSTM is a novel recurrent neural‑network architecture that extends LSTM with exponential gating and a matrix memory, enabling large‑scale language models (e.g., a 7 B‑parameter model). The repo provides PyTorch code, custom CUDA/Triton kernels (via the `mlstm_kernels` package), a single‑file 7B implementation, configuration utilities, and pretrained weights on Hugging Face. Installation is via conda + pip, and the library works on NVIDIA GPUs (fast Triton kernels) with fallbacks for AMD or Apple Silicon.