NVlabs/OmniVinci

OmniVinci is an omni-modal LLM for joint understanding of vision, audio, and language.

What it solves

OmniVinci is designed to improve the joint understanding of multiple modalities—specifically vision, audio, and text—within a single Large Language Model (LLM). It addresses the challenge of aligning different sensory inputs (like sound and sight) so the model can reason across them simultaneously, which is critical for applications in robotics, medical AI, and smart factories.

How it works

OmniVinci utilizes a specialized architecture and a curated dataset of 24 million single-modal and omni-modal conversations to achieve its goals. It introduces three primary architectural innovations:

  • OmniAlignNet: Strengthens the alignment between vision and audio embeddings in a shared latent space.
  • Temporal Embedding Grouping: Captures the relative temporal alignment between vision and audio signals.
  • Constrained Rotary Time Embedding: Encodes absolute temporal information into the omni-modal embeddings.

Who it’s for

This project is intended for AI researchers and developers building omni-modal systems that require high-performance cross-modal understanding, particularly those working with video and audio integration.

Highlights

  • High Efficiency: Outperforms Qwen2.5-Omni on several benchmarks (DailyOmni, MMAR, Video-MME) while using 6 times fewer training tokens (0.2T vs 1.2T).
  • Omni-Modal Support: Jointly processes vision, audio, and text.
  • Open Source: Provides an open-source model (OmniVinci-9B) and codebase.
  • Broad Application: Demonstrated utility in robotics, medical AI, and smart factory environments.

Related

  • Project
    IamCreateAI/NeoVerse

    NeoVerse is a 4D world model that reconstructs 3D scenes from monocular videos to enable high-quality video generation along novel camera trajectories.

  • Project
    uezo/aiavatarkit

    A modular framework for building low-latency conversational AI avatars with integrated VAD, STT, LLM, and TTS pipelines across multiple channels.

  • Project
    Osilly/Vision-DeepResearch

    A framework and benchmark for developing Multimodal Large Language Models capable of deep research through iterative visual and textual search across images and videos.

  • Project
    TIGER-AI-Lab/VLM2Vec

    A unified framework for training and evaluating omni-modality embedding models that produce fixed-dimensional vectors for text, images, videos, audio, and visual documents.