OpenSenseNova/SenseNova-Vision

What it solves

SenseNova-Vision addresses the fragmentation of computer vision tasks by formulating them as a unified multimodal generation problem. Instead of using separate models or task-specific prediction heads for different tasks (like object detection, segmentation, and depth estimation), it uses a single unified multimodal model (UMM) to handle diverse visual tasks through native text and image generation.

How it works

The system treats visual tasks as instruction-response pairs. It uses natural-language instructions and visual prompts to specify the task and output format. The model then generates responses in three formats:

• Text generation: Used for symbolic records such as bounding boxes, categories, OCR strings, and camera parameters.
• Image generation: Used for dense spatial targets like segmentation masks, depth maps, and surface normals.
• Mixed text-image generation: Used for compositional tasks that require both symbolic and dense outputs.

To achieve this, the project introduces the SenseNova-Vision Corpus, a large-scale dataset of computer-vision instruction-response examples, and trains a pretrained UMM without requiring architectural branches or specialized decoders.

Who it’s for

• AI researchers and developers working on generalist vision models.
• Developers needing a single model to perform multiple CV tasks (detection, segmentation, geometry) without switching models.
• Users interested in multi-view visual geometry and 3D reconstruction.

Highlights

• Unified Formulation: Casts heterogeneous CV tasks into a single input-output space.
• Broad Capability: Supports structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry.
• No Task-Specific Heads: Operates without the need for specialized prediction heads or decoders.
• Lage-Scale Corpus: Includes the SenseNova-Vision Corpus-50M for training.