facebookresearch/tuna-2

Official implementation of Tuna-2: Pixel Embeddings Beat Vision Encoders for Unified Understanding and Generation

What it solves

Tuna-2 addresses the complexity of unified multimodal models (UMMs) that typically rely on heavy visual encoding components like Variational Autoencoders (VAEs) or representation encoders to process images. It aims to simplify the architecture while improving performance across both understanding and generation tasks.

How it works

The project evolves the architecture by progressively removing visual encoding layers. While the original Tuna used a VAE, and Tuna-R used a representation encoder, Tuna-2 removes both, utilizing direct patch embedding layers to handle raw image inputs as pixel embeddings. This streamlined approach allows the model to process visual data more directly, outperforming its predecessors in multimodal benchmarks.

Who it’s for

Researchers and developers working on unified multimodal models, specifically those interested in efficient image generation (text-to-image), image editing, and the development of native multimodal architectures that bypass traditional vision encoders.

Highlights

  • Unified Architecture: Handles both understanding and generation within a single framework.
  • Pixel-Space Processing: Replaces complex vision encoders with direct patch embeddings for raw images.
  • Versatile Tasks: Supports text-to-image generation and image editing.
  • Extensible to Video: Includes the complete codebase for training and inference of video generation models.

Related

  • Project
    EvolvingLMMs-Lab/LLaVA-OneVision-2

    A fully open 8B multimodal model that unifies image, long-form video, and spatial understanding using a codec-aligned vision encoder for efficient long-video reasoning.

  • Project
    OpenSenseNova/SenseNova-U1

    SenseNova-U1 is a native multimodal model series that unifies understanding, reasoning, and generation within a single architecture, eliminating the need for modality adapters.

  • Project
    OpenSenseNova/SenseNova-Vision

    SenseNova-Vision is a unified multimodal model that treats diverse computer vision tasks as a generation problem, producing text, images, or mixed outputs for tasks like detection, segmentation, and depth estimation.

  • Project
    UCSC-VLAA/OpenVision

    A family of open-source, cost-effective vision encoders for multimodal learning that optimizes training efficiency and performance across understanding and generation tasks.

  • Project
    Tencent/WeMM-Embedding

    A family of universal multimodal embedding models that provide unified representations for text, images, videos, and visual documents to enable high-performance cross-modal retrieval.