kyegomez/Gemini

What it solves

This project provides an open-source implementation of a multimodal transformer model inspired by Google's Gemini. It aims to create a system capable of processing and generating multiple modalities—including text, images, and audio—within a single transformer architecture rather than relying on separate encoders for each.

How it works

The model takes text, audio, images, and videos as input sequences and transforms them into tokens. Unlike traditional visual transformers (ViT) that use a separate encoder, this implementation feeds image embeddings directly into the transformer. It uses special modality tokens (such as [IMG] or [AUDIO]) to distinguish between input types. The architecture incorporates several optimization techniques, including Flash Attention, QK norm, and Rotary Positional Embeddings (RoPE), and includes a LongGemini variant that utilizes Ring Attention for extended context.

Who it’s for

Developers and researchers interested in building or training multimodal AI models that can natively handle interleaved text, image, and audio data.

Highlights

• Native Multimodality: Processes text, images, and audio simultaneously in one transformer.
• Optimized Performance: Implements Flash Attention, QK norm, and KV caching for efficient inference.
• Extended Context: Includes a LongGemini implementation using Ring Attention.
• Custom Tokenizer: Features a MultimodalSentencePieceTokenizer based on the Llama tokenizer with added modality-specific tokens.