kyegomez/VisionMamba

Implementation of Vision Mamba from the paper: "Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model" It's 2.8x faster than DeiT and saves 86.8% GPU memory when performing batch inference to extract features on high-res images

What it solves

It provides an efficient alternative to traditional Vision Transformers (ViTs) for visual representation learning. It addresses the high computational cost and memory usage associated with standard attention mechanisms when processing high-resolution images.

How it works

The project implements a Bidirectional State Space Model (SSM) to process visual data. Unlike standard transformers, this architecture allows for more efficient feature extraction, significantly reducing the GPU memory footprint and increasing inference speed.

Who it’s for

Researchers and developers working on computer vision tasks that require high-resolution image processing with limited GPU memory or a need for higher inference speeds.

Highlights

  • 2.8x faster than DeiT during batch inference.
  • Reduces GPU memory usage by 86.8% when extracting features from high-res images.
  • Easy installation via pip.

Related

  • Project
    kyegomez/MultiModalMamba

    A multi-modal AI model that integrates Vision Transformer (ViT) and Mamba to efficiently process and interpret text, images, audio, and video data simultaneously.

  • Project
    state-spaces/mamba

    Mamba is a linear-time sequence modeling architecture based on selective state space models (SSMs) that provides a high-performance, subquadratic alternative to Transformers.

  • Dispatch
    Falcon Mamba 7B Release Notes

    The Technology Innovation Institute (TII) has released Falcon Mamba 7B, the first large-scale pure State Space Language Model (SSLM) that matches the performance of state-of-the-art transformer models while eliminating attention-based memory scaling issues.

  • Dispatch
    Bamba-9B: Inference-Efficient Hybrid Mamba2 Model

    Bamba-9B, a hybrid Mamba2 model from IBM, Princeton, CMU, and UIUC trained on 2.2T open tokens, achieves 2.5x throughput and 2x latency improvements over Llama 3.1 8B in vLLM and is immediately usable in transformers, vLLM, TRL, and llama.cpp.

  • Project
    modelscope/DiffSynth-Studio

    An open-source Diffusion model engine that enables the inference and training of image, video, and audio generative models on consumer-grade GPUs through advanced VRAM management and quantization.