facebookresearch/VLM3

Official implementation of paper "VLM³: Vision Language Models Are Native 3D Learners".

What it solves

VLM³ addresses the complexity of 3D vision tasks. Traditionally, achieving high performance in 3D understanding requires specialized architectures, complex loss functions, and extensive data augmentation. VLM³ demonstrates that standard Vision Language Models (VLMs) can perform these tasks effectively without these specialized additions.

How it works

The project uses a standard VLM architecture (specifically based on Qwen3-vl-4B) and trains it using Supervised Fine-Tuning (SFT) with a unified text-based output domain. To handle 3D tasks, it employs two key techniques:

  1. Focal Length Normalization: Input images are resized so that the focal length is consistent across all images, removing camera ambiguity without needing extra encoders.
  2. Normalized Coordinate Referencing: The model refers to objects or pixels using text-based normalized coordinates (e.g., [0, 2000) or [0, 1000)) for both axes, eliminating the need for marker rendering or architectural changes.

Who it’s for

It is designed for researchers and developers working on 3D vision, depth estimation, and camera pose estimation who want a more scalable and simpler approach to 3D learning using foundation models.

Highlights

  • No Architecture Changes: Works with standard VLM architectures and losses without needing custom prediction heads or routings.
  • High Performance: Matches or surpasses state-of-the-art models in metric depth estimation, pixel correspondence, and camera pose estimation.
  • Simplified Pipeline: Removes the need for complex data augmentations like rotation, translation, or appearance augmentation.
  • Unified Output: Uses a simple text-based output for 3D spatial reasoning.

Related

  • Dispatch
    Vision Language Models Explained

    Hugging Face provides a comprehensive guide to Vision Language Models (VLMs), detailing their architecture, open-source options, evaluation benchmarks, and new experimental support for fine-tuning via the TRL library.

  • Project
    MCG-NJU/VideoChat3

    VideoChat3 is a 4B generalist video MLLM designed for efficient video understanding, covering everything from subtle motion and long-form reasoning to live stream proactive responses.

  • Project
    bytedance/Sa2VA

    A family of projects, anchored by Sa2VA, that integrates SAM-2 with multimodal LLMs to enable pixel-level grounded understanding and segmentation of images and videos.

  • Project
    om-ai-lab/VLM-R1

    VLM-R1 is a framework for training R1-style Large Vision-Language Models using GRPO to improve reasoning and generalization in tasks like visual grounding, math, and object detection.

  • Dispatch
    nanoVLM: A Minimalist PyTorch Toolkit for Training Vision Language Models

    Hugging Face has released nanoVLM, a lightweight, pure PyTorch toolkit designed to simplify the training and understanding of Vision Language Models (VLMs) for beginners and researchers.