BlinkDL/RWKV-LM

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

What it solves

RWKV is designed to combine the high performance of Transformer-based Large Language Models (LLMs) with the efficiency of Recurrent Neural Networks (RNNs). It eliminates the need for a KV-cache, providing linear-time complexity and constant-space memory usage during inference, which solves the VRAM and speed bottlenecks associated with traditional attention mechanisms.

How it works

RWKV is a 100% attention-free RNN architecture. It can be trained in parallel like a GPT-style Transformer (GPT mode) but operates as a standard RNN during inference (RNN mode), requiring only the hidden state of the current token to compute the next. The latest version, RWKV-7 "Goose", acts as a meta-in-context learner that uses in-context gradient descent at every token to update its state.

Who it’s for

This project is for AI researchers and developers who want to deploy LLMs with significantly lower memory overhead, achieve faster inference speeds, and handle potentially "infinite" context lengths without the quadratic scaling costs of attention.

Highlights

  • Linear-Time & Constant-Space: No KV-cache required, ensuring constant speed and VRAM usage regardless of sequence length.
  • Parallelizable Training: Can be trained like a Transformer for high efficiency on GPUs.
  • Meta-In-Context Learning: RWKV-7 utilizes test-time training to update its state on the context.
  • High Efficiency: Supports extremely fast decoding speeds (e.g., 10,000+ tokens/s for 7.2B model on RTX 5090 with high batch sizes).
  • Open Source: A Linux Foundation AI project that is totally free.

Related

  • Project
    BlinkDL/ChatRWKV

    A chat interface and inference implementation for the RWKV language model, an RNN-based alternative to Transformers that offers similar quality with higher speed and lower VRAM usage.

  • Dispatch
    RWKV Architecture Integration into Hugging Face Transformers

    Hugging Face announced the integration of the RWKV RNN‑Transformer hybrid model into the Transformers library, enabling open‑source, long‑context language models that combine RNN efficiency with transformer performance.

  • Project
    Zefan-Cai/R-KV

    R-KV is a redundancy-aware KV-cache compression method for reasoning models that reduces memory usage by up to 90% and boosts throughput without losing accuracy.

  • Project
    rednote-machine-learning/RedKnot

    RedKnot is a model-aware execution framework that optimizes long-context LLM serving by using head-aware KV reuse, sparse FFN/MoE execution, and SegPagedAttention to reduce TTFT and compute costs.

  • Project
    josStorer/RWKV-Runner

    A lightweight, automated runner for RWKV models that provides an OpenAI-compatible API and a user-friendly interface for chat, completion, and music composition.