Anionex/agent-vision-toolkit

What it solves

It enables text-only LLM agents (such as those powered by DeepSeek) to perform visual tasks by providing them with a set of external vision tools and a structured methodology. This bypasses the limitation of models that lack native multimodality or have system-blocked image tools, allowing them to handle image Q&A, UI restoration, and GUI automation as effectively as native multimodal agents.

How it works

Instead of generating generic image descriptions, the toolkit uses a "focus hint" approach. It extracts the agent's current intent from the conversation and passes it to a multimodal vision model as a hint, ensuring the resulting description is task-aware and specific to the current goal. It provides this capability through two main components:

1. Vision tool CLIs: A set of command-line tools (glance, ground, detect, trace, crop) that any agent with shell access can invoke.
2. Seamless Integration: A local proxy or native plugins for specific agents (like Codex, Claude Code, and OpenCode) that allow pasted images and built-in image tools to work transparently.

Who it’s for

Developers and users of text-only coding agents who want to add visual capabilities like screenshot analysis, UI reconstruction from sketches, or autonomous GUI operation without switching to a native multimodal model.

Highlights

• Task-Aware Vision: Captures specific details the LLM cares about based on current intent rather than providing broad descriptions.
• Linguistic Flexibility: Supports multiple vision API protocols including OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages.
• Specialized Visual Tools: Includes tools for bounding box localization (ground), element inventory (detect), and deterministic SVG trajectory recovery (trace).
• Ready-made Playbooks: Provides step-by-step guides for complex tasks like long-screenshot OCR, UI restoration from designs, and GUI automation.