om-ai-lab/VLX-Seek

TL;DR

VLX‑Seek is an open‑source vision‑language model (VLM) that lets edge‑device agents (drones, robots, cameras, etc.) understand what is in an image and precisely where it is. Instead of asking the model to output raw bounding‑box numbers, VLX‑Seek first generates a set of candidate regions, turns each region into a special “region token”, and then lets the language model select or refer to those tokens. The result is a compact, token‑based answer that can be mapped back to actual boxes for detection, referring‑expression comprehension, OCR, counting, and other fine‑grained perception tasks.

---

What the repo contains

Item	Description
inference.py	Command‑line demo that loads the 10 B VLX‑Seek checkpoint, runs an optional region detector (WeDetect), and prints JSON + visual results for detection or VQA.
vlx_seek/ (the vlx_seek package)	Model architecture, the hybrid fine‑grained region encoder (HFRE), token‑to‑region mapping, and the VLXSeekWorker Python wrapper.
requirements.txt	Python‑3.10+ dependencies (PyTorch, transformers, etc.).
docs/	Detailed inference guide, supported tasks, and API reference.
demo/	Sample image used in the README.
assets/	Logos, example result screenshots, and community QR codes.
resources/ (optional)	Place to store the downloaded VLX‑Seek checkpoint and the WeDetect detector weights.

Core ideas

1. Region‑first pipeline – a lightweight detector proposes boxes; each box becomes a region token (<obj0>, <obj1>, …).
2. Hybrid Fine‑Grained Region Encoder (HFRE) – merges a high‑level semantic pathway (the base VLM) with a detail‑rich local pathway, producing embeddings that live in the same space as the LLM tokens.
3. Token‑based reasoning – the language model receives the image tokens, the text query, and the region tokens, then answers by emitting region IDs (e.g., <obj2><obj5>) instead of numeric coordinates. This is shorter, easier to parse, and aligns with how LLMs naturally select entities.
4. Open‑vocabulary & rejection – the model can answer “none” when no region matches, reducing hallucinated detections.

What you can do with it

• Open‑vocabulary detection – ask for arbitrary objects (e.g., "orange; apple").
• Referring expression comprehension – locate a specific instance described in natural language.
• Region OCR / captioning – read text or generate detailed captions for selected boxes.
• Counting – return the number of matching regions.
• General VQA – answer questions about the whole image without any proposals.
• Multi‑step visual reasoning – because region tokens are ordinary language tokens, the model can compare, contrast, and explain its choices.

Getting started (quick‑start)

# 1. Clone and install
git clone https://github.com/om-ai-lab/VLX-Seek.git
cd VLX-Seek
pip install -r requirements.txt

# 2. Run detection (proposals are generated automatically)
python inference.py \
  --image-path demo/demo_image.jpg \
  --task detection \
  --text "orange; apple"

# 3. Run VQA (no proposals needed)
python inference.py \
  --image-path demo/demo_image.jpg \
  --task vqa \
  --text "What fruits are in the image?"

The first run will download the 10 B checkpoint from Hugging Face (omlab/VLX-Seek-1.5-10B) and the WeDetect detector (wedetect_base_uni.pth). Results are printed as JSON and saved as <image>_result.png.

Python API (for developers)

from vlx_seek_worker import VLXSeekWorker

worker = VLXSeekWorker("omlab/VLX-Seek-1.5-10B", device="cuda")

# detection – let the built‑in detector propose boxes
out = worker.run(
    image_path="demo/demo_image.jpg",
    task="detection",
    text="orange; apple"
)
print(out["region_refs"])   # e.g. ["<obj2>", "<obj5>"]
print(out["boxes"])        # actual [x1,y1,x2,y2] coordinates

The run method also accepts a pre‑computed bbox_list if you want to supply your own proposals.

Model size & performance

• VLX‑Seek 1.5‑10B – 10 B parameters, released with inference code. Smaller 0.6 B and 3 B variants are planned.
• Faster inference – linear‑attention layers and a streamlined proposal pipeline reduce latency on edge GPUs.
• Reduced hallucination – hard‑negative training and an explicit None output format improve the model’s ability to say no target found.

Where to find more

• Model checkpoint – https://huggingface.co/omlab/VLX-Seek-1.5-10B
• Blog & architecture details – https://om-ai-lab.github.io/2026_07_06_vlx_seek_1_5_en.html
• Demo – https://om-agent.com/#/front (interactive web demo)
• Community – Discord, WeChat QR codes in the README; contact marketing@hzlh.com for partnerships.

---

Bottom line

VLX‑Seek turns visual regions into first‑class language tokens, letting a large language model reason about objects instead of merely spitting out coordinate strings. This design is especially useful for low‑power, real‑time embodied systems that need reliable, fine‑grained grounding while keeping inference fast and parsable.