OpenSenseNova/SenseNova-Vision

What it solves

SenseNova-Vision addresses the fragmentation of computer vision tasks by formulating them as a unified multimodal generation problem. Instead of using separate models or specialized prediction heads for different tasks (like object detection, segmentation, or depth estimation), it uses a single Unified Multimodal Model (UMM) to handle diverse visual tasks through native text and image generation.

How it works

The system treats visual tasks as instruction-response pairs. It uses natural-language instructions and optional visual prompts to define the task. The model then generates responses in three formats:

• Text generation: Used for symbolic records such as bounding boxes, points, OCR strings, and camera parameters.
• Image generation: Used for dense spatial targets like segmentation masks, depth maps, and surface normals.
• Mixed responses: Combines both text and image outputs for complex compositional tasks.

To achieve this, the project introduces the SenseNova-Vision Corpus, a large-scale dataset of computer-vision instruction-response examples, and trains the model starting from a pretrained UMM without requiring task-specific architectural branches.

Who it’s for

This project is for AI researchers and developers working on generalist vision models, multimodal LLMs, and those needing a single model capable of performing structured visual understanding, dense geometric prediction, and segmentation.

Highlights

• Unified Formulation: Casts heterogeneous CV tasks into the native input-output spaces of UMMs.
• Diverse Capabilities: Supports object detection, OCR, GUI understanding, keypoint detection, monocular depth estimation, surface normal estimation, and multi-view reconstruction.
• No Specialized Heads: Operates without task-specific prediction heads or decoders.
• Large-scale Corpus: Includes the SenseNova-Vision Corpus containing 50M examples.