BlinkDL/ChatRWKV

ChatRWKV is like ChatGPT but powered by RWKV (100% RNN) language model, and open source.

What it solves

ChatRWKV provides a way to run a high-quality language model that matches Transformer performance while being significantly faster and using less VRAM. It addresses the efficiency bottlenecks of traditional Transformers by using a Recurrent Neural Network (RNN) architecture.

How it works

It is powered by the RWKV language model, a 100% RNN that scales like a Transformer. The project includes a chat interface and inference scripts that allow users to interact with the models. It supports various inference strategies (such as "stream" and "split") and quantization (INT8) to further reduce memory requirements, enabling large models (like the 14B version) to run on consumer hardware with as little as 3GB of VRAM.

Who it’s for

Developers and AI enthusiasts who want to run powerful LLMs locally with lower hardware requirements or those looking to build their own inference engines based on the RNN architecture.

Highlights

  • RNN Efficiency: Matches Transformer quality but with faster speeds and lower memory usage.
  • Flexible Inference: Supports multiple strategies and INT8 quantization for low VRAM usage.
  • CUDA Acceleration: Includes optional CUDA kernels for significantly faster sequence mode processing.
  • Developer Friendly: Provides simplified scripts (like "RWKV in 150 lines") and API demos to help developers understand and implement the model.

Related

  • Project
    BlinkDL/RWKV-LM

    An attention-free RNN architecture that delivers Transformer-level LLM performance with linear-time complexity and constant VRAM usage.

  • Dispatch
    RWKV Architecture Integration into Hugging Face Transformers

    Hugging Face announced the integration of the RWKV RNN‑Transformer hybrid model into the Transformers library, enabling open‑source, long‑context language models that combine RNN efficiency with transformer performance.

  • Project
    alibaba/rtp-llm

    A high-performance LLM inference acceleration engine developed by Alibaba that optimizes throughput and latency for production-scale deployments.

  • Project
    foldl/chatllm.cpp

    A C++ LLM inference engine based on ggml that enables memory-efficient local execution of multimodal models with RAG and quantization support.

  • Project
    jatinchowdhury18/RTNeural

    A lightweight C++ neural network inferencing engine optimized for real-time systems and audio processing.