Anionex/agent-vision-toolkit

为纯文本模型"看图“设计更好的视觉工具箱和技能，支持多图理解，图片问答，前端UI还原、GUI 自动化等，并可选无缝接入多个主流agent，直接识别粘贴图片｜ A vision toolkit and skill designed for text-only llms — image Q&A, long-screenshot OCR, frontend UI restoration, and GUI automation, with optional seamless integration for Codex, Claude Code, Pi, Oh My Pi, and OpenCode

What it solves

agent-vision-toolkit provides a way to give text-only LLM agents (such as those running on DeepSeek) visual capabilities. It allows these agents to perform image Q&A, OCR on long screenshots, UI restoration, and GUI automation without requiring the underlying model to be natively multimodal.

How it works

The toolkit implements vision capabilities in the "harness" rather than the model. It uses a combination of:

  • Vision tool CLIs: A set of command-line tools (glance, ground, detect, trace, crop) that an agent can invoke via a shell to analyze images, locate objects, and extract graphics.
  • Intent-aware descriptions: Instead of generic image descriptions, the toolkit extracts the agent's current intent from the conversation and passes it as a focus hint to a multimodal API, ensuring the resulting description is relevant to the task.
  • Seamless Integration: Optional local proxies or native plugins for specific agents (like Codex, Claude Code, Pi, and OpenCode) that allow pasted images and built-in image tools to work transparently.
  • Agent Skills: A vision-skills package that teaches the agent how to sequence these tools to solve complex visual tasks.

Who it’s for

Developers and users of text-only coding agents who want to add multimodal capabilities like UI reconstruction from sketches or autonomous GUI operation using models that lack native vision support.

Highlights

  • Task-focused vision: Captures details based on the model's actual intent rather than providing broad, unfocused descriptions.
  • Both pasted images and built-in tools work: Supports direct image pasting and the agent's own internal image tools through a proxy layer.
  • Battle-tested workflows: Includes playbooks for long-screenshot OCR, UI restoration from sketches, and GUI automation.
  • One-sentence installation: Designed to be installed by the agent itself via a simple prompt.
  • Local SVG tracing: Deterministically recovers geometric trajectories from graphics to create editable SVGs without LLM participation.

Related

  • Project
    Anionex/dsh-vision-toolkit

    A vision toolkit for DeepSeek Harness that gives text-only agents the ability to perform image Q&A, UI restoration, and GUI automation through a combination of vision models and local processing tools.

  • Project
    askui/python-sdk

    A Python SDK that enables AI agents to control desktop and mobile devices using vision-based automation instead of brittle UI selectors.

  • Project
    jmerelnyc/Photo-agents

    An autonomous, vision-grounded agent framework that allows LLMs to perceive and operate a computer screen through a perceive-reason-act loop.

  • Dispatch
    DeepSeek Introduces Vision Capabilities

    DeepSeek has integrated vision capabilities into its chat interface, enabling the model to understand and describe image content rather than just performing text extraction.

  • Project
    zai-org/GLM-V

    A series of vision-language models (VLMs) that enhance multimodal reasoning, supporting native function calling, visual grounding, and complex document understanding.