om-ai-lab/VLX-Seek

VLX-Seek is an efficient VLM that grounds fine-grained visual targets with high precision in wild environments.

What it solves

VLX-Seek addresses the fragility and inefficiency of traditional Vision-Language Models (VLMs) that rely on generating long numeric coordinate strings (e.g., [x1, y1, x2, y2]) for object localization. Such coordinate-based methods are prone to formatting errors, hallucinate boxes when targets are absent, and slow down inference due to the length of the output sequences.

How it works

Instead of generating coordinates, VLX-Seek reformulates localization as a region retrieval and reference task. It uses a decoupled pipeline:

  1. Region Proposal: A candidate region generation network (such as the integrated WeDetect-Base-Uni) proposes potential foreground regions in an image.
  2. Region Tokens: These candidate regions are encoded as addressable region tokens (e.g., <obj0>, <obj1>).
  3. Region Reference: The language model identifies and selects the specific region tokens that match the user's query, outputting short region IDs rather than coordinates. These IDs are then mapped back to the original bounding boxes.

Who it’s for

It is designed for edge-side embodied vision systems where precise spatial grounding is critical, such as drones, monitoring cameras, robots, and robot dogs, especially those operating under limited compute and power budgets.

Highlights

  • Region-based Localization: Replaces brittle coordinate generation with a more stable region-referencing mechanism.
  • Embodied Perception: Specifically trained on drone, surveillance, and robot-view data for better performance in real-world embodied scenarios.
  • Hallucination Reduction: Includes hard-negative rejection training and an explicit None format to avoid detecting absent targets.
  • Efficiency: Faster inference and lower memory costs through a faster proposal pipeline and Linear Attention layers.

Related

  • Dispatch
    Hugging Face Vision Language Models 2025 Update

    Hugging Face provides a comprehensive overview of the 2024-2025 evolution of Vision Language Models (VLMs), highlighting trends in any-to-any architectures, reasoning models, and the rise of Vision-Language-Action (VLA) models for robotics.

  • Project
    TIGER-AI-Lab/Pixel-Reasoner

    Pixel Reasoner is a research framework that adds visual‑manipulation operations (zoom‑in, select‑frame, etc.) to a Vision‑Language Model, training it via instruction‑tuning and curiosity‑driven reinforcement learning. The repo supplies installation guides, scripts for both SFT and RL stages, pretrained 7 B checkpoints, datasets, and evaluation pipelines for image and video benchmarks.

  • Dispatch
    Vision Language Models Explained

    Hugging Face provides a comprehensive guide to Vision Language Models (VLMs), detailing their architecture, open-source options, evaluation benchmarks, and new experimental support for fine-tuning via the TRL library.

  • Dispatch
    Qwen2-VL release: open-source 2B/7B vision-language models and 72B API with state-of-the-art image, video, and multilingual capabilities

    Qwen released Qwen2-VL, a new vision-language model series (2B, 7B open-source, 72B API) that sets state-of-the-art performance on image, document, multilingual, and long-video understanding while supporting visual agent capabilities.

  • Project
    tiiuae/Falcon-Perception

    Falcon‑Perception is an open‑source, PyTorch/MLX inference library for the Falcon multimodal transformer that can answer natural‑language queries with object detection, instance segmentation, or OCR. It provides highly‑optimized paged and batch engines, a FastAPI server, a Streamlit UI, Docker/vLLM deployment for OCR, and Colab notebooks. The 270 M‑parameter Falcon‑OCR 1.5 model delivers end‑to‑end OCR quality comparable to much larger VLMs while staying three‑times smaller.