state-spaces/mamba

Mamba SSM architecture

What it solves

Mamba addresses the limitations of Transformers in processing information-dense data, specifically the quadratic scaling issues that make long-sequence modeling inefficient. It provides a subquadratic, linear-time alternative for sequence modeling that maintains high performance on tasks like language modeling.

How it works

Mamba is based on a selective State Space Model (SSM) architecture. It utilizes a hardware-aware design and efficient CUDA implementations—similar to the spirit of FlashAttention—to optimize performance. The project has evolved through several iterations:

  • Mamba-1: Introduced selective state spaces for linear-time sequence modeling.
  • Mamba-2: Introduced Structured State Space Duality (SSD), framing Transformers as SSMs to create more efficient algorithms.
  • Mamba-3: Further improved sequence modeling using state space principles and an inference-first approach.

Who it’s for

It is designed for AI researchers and developers building large-scale sequence models or language models who need the efficiency of linear-time scaling without sacrificing the performance typically associated with Transformers.

Highlights

  • Linear-Time Scaling: Offers a more efficient alternative to the quadratic complexity of Transformers.
  • Hardware-Aware Design: Includes optimized CUDA extensions (selective_scan_cuda) for high-performance GPU execution.
  • Pretrained Models: Provides a range of pretrained base models (from 130M to 2.8B parameters) trained on the Pile and SlimPajama datasets.
  • Flexible Interface: Exposes multiple levels of API, from raw selective SSM layers to full Mamba blocks and complete language model examples.

Related

  • Dispatch
    Codestral Mamba Release Notes

    Mistral AI has released Codestral Mamba, a 7B parameter model utilizing the Mamba architecture to provide linear time inference and high-performance coding and reasoning capabilities.

  • Project
    kyegomez/MultiModalMamba

    A multi-modal AI model that integrates Vision Transformer (ViT) and Mamba to efficiently process and interpret text, images, audio, and video data simultaneously.

  • Dispatch
    Falcon Mamba 7B Release Notes

    The Technology Innovation Institute (TII) has released Falcon Mamba 7B, the first large-scale pure State Space Language Model (SSLM) that matches the performance of state-of-the-art transformer models while eliminating attention-based memory scaling issues.

  • Project
    kyegomez/VisionMamba

    An implementation of Vision Mamba, a bidirectional state space model for efficient visual representation learning that is significantly faster and more memory-efficient than traditional Vision Transformers.

  • Project
    NX-AI/xlstm

    A new Recurrent Neural Network architecture that extends LSTM to compete with Transformers and State Space Models, featuring a 7B parameter language model for efficient inference.