om-ai-lab/VLM-FO1

VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMs

What it solves

Existing Vision-Language Models (VLMs) often struggle with fine-grained perception—the ability to precisely locate and understand small or specific details in an image—while maintaining their general high-level reasoning capabilities. VLM-FO1 bridges this gap by adding precise spatial awareness to pre-trained models without erasing their general knowledge.

How it works

VLM-FO1 is a plug-and-play module that integrates with any existing VLM. It uses a Hybrid Region Encoder (HFRE) with a Dual-Vision Encoder architecture that blends semantic-rich features with perception-enhanced features. This creates "region tokens" that capture both the overall meaning and the fine-grained spatial details of specific areas. To prevent the model from forgetting its original general abilities, it employs a two-stage training strategy.

Who it’s for

It is designed for researchers and developers building perception-aware AI models that need to perform tasks like object grounding, region-based reasoning, and precise object counting.

Highlights

  • Plug-and-Play: Can be added to any pre-trained VLM without altering the original weights.
  • Fine-Grained Tasks: Excels at object grounding, region generative understanding, and visual region reasoning.
  • Flexible Integration: Works with external object detectors like UPN or segmentation tools like SAM3 for high-fidelity detection and video tracking.
  • Preservation of Knowledge: Specifically trained to avoid "catastrophic forgetting" of the base model's general visual understanding.

Related

  • Project
    om-ai-lab/VLM-R1

    VLM-R1 is a framework for training R1-style Large Vision-Language Models using GRPO to improve reasoning and generalization in tasks like visual grounding, math, and object detection.

  • Project
    Wakals/CoVT

    CoVT is a framework that enables Vision-Language Models to reason using continuous visual tokens, improving their spatial reasoning and geometric awareness by grounding semantic thoughts in perceptual cues.

  • Dispatch
    Hugging Face Vision Language Models 2025 Update

    Hugging Face provides a comprehensive overview of the 2024-2025 evolution of Vision Language Models (VLMs), highlighting trends in any-to-any architectures, reasoning models, and the rise of Vision-Language-Action (VLA) models for robotics.

  • Project
    zai-org/GLM-V

    A series of vision-language models (VLMs) that enhance multimodal reasoning, supporting native function calling, visual grounding, and complex document understanding.

  • Project
    gokayfem/ComfyUI_VLM_nodes

    A production-oriented suite of ComfyUI nodes for vision-language models, providing structured detection, segmentation, and adaptive video reasoning with VRAM optimizations.