facebookresearch/VLM3

What it solves

VLM³ addresses the complexity of 3D vision tasks. Traditionally, achieving state-of-the-art results in 3D understanding required specialized architectures, complex loss functions, and extensive data augmentations. VLM³ demonstrates that standard Vision Language Models (VLMs) can perform these tasks effectively without these specialized additions.

How it works

The project uses a standard VLM architecture (specifically Qwen3-vl-4B) and trains it using Supervised Fine-Tuning (SFT) with a unified text-based output domain. To handle 3D spatial data:

• Camera Ambiguity: It resizes input images to a consistent focal length (e.g., 1000 pixels) to remove the need for extra encoders.
• Spatial Referencing: It refers to objects or pixels using normalized text coordinates (e.g., [0, 2000) or [0, 1000)) for both axes, eliminating the need for marker rendering or architectural changes.

Who it’s for

Researchers and developers in 3D computer vision and multimodal AI who want a simpler, more scalable way to implement 3D learning without designing task-specific models.

Highlights

• No Architectural Changes: Works with standard VLM architectures and loss functions.
• No Complex Augmentation: Eliminates the need for image cropping, rotation, or translation augmentations.
• Broad 3D Capability: Performs metric depth estimation, pixel correspondence estimation, and camera pose estimation.
• Competitive Performance: Matches or surpasses specialized models like UnidepthV2, Moge-2, and DepthAnything3 in various 3D benchmarks.