Wakals/CoVT

What it solves

Vision-Language Models (VLMs) often struggle with dense visual perception tasks—such as spatial reasoning, geometric awareness, and precise object localization—because they primarily reason in a discrete linguistic space. CoVT addresses this by allowing models to reason using continuous visual tokens, bridging the gap between high-level semantic reasoning and low-level perceptual grounding.

How it works

CoVT introduces a "Chain-of-Visual-Thought" where the VLM predicts a small set of compact, continuous visual tokens (roughly 20) before generating a final text answer. These tokens are trained to encode specific perceptual cues by aligning them with lightweight vision experts during training:

• Segmentation (SAM): 8 tokens for mask prompts.
• Depth (DepthAnything): 4 tokens for depth reconstruction.
• Edges (PIDINet): 4 tokens for edge structure.
• Features (DINO): 4 tokens for patch-level features.

During inference, the model reasons directly in this latent visual space. While these tokens can be decoded into dense predictions (like depth maps or segmentation masks) for human interpretability, this decoding step is optional and not required for the model to reach its answer.

Who it’s for

Researchers and developers working on multimodal AI who need VLMs with improved spatial precision, better geometric understanding, and more interpretable reasoning processes for vision-centric tasks.

Highlights

• Continuous Visual Reasoning: Moves beyond text-only thought chains to include latent visual representations.
• Cues from Experts: Leverages specialized models (SAM, DINO, etc.) to distill perceptual knowledge into compact tokens.
• Performance Boost: Improves performance by 3% to 16% on benchmarks like CV-Bench, MMVP, and RealWorldQA.
• Interpretability: Allows the internal "visual thoughts" to be decoded into viewable maps for debugging or verification.