ysr666/dsh-vision-router

dsh‑vision‑router – “eyes” for DeepSeek Harness

What it is

• A DeepSeek Harness (DSH) plugin that lets text‑only agents see images without losing any pixel information. Instead of converting an image to a plain text description, the plugin routes the raw image to a vision model (or a chain of free vision back‑ends) and returns structured results that the LLM can call as ordinary tools.

Why it matters

• Most existing DSH vision plugins only provide a lossy description of an image. dsh‑vision‑router keeps the original pixels on the vision side, so the LLM can ask follow‑up questions, crop, compare, or run OCR on the exact same picture.
• It works out‑of‑the‑box, free, and without any Python dependency – everything runs on Node ≥ 22 using sharp, potrace, tesseract and a system Chrome.

Key features

Feature	What you get
One‑command install	npx @deepseek-ai/dsh plugin --profile web add dsh-vision-router (or via dshpm) automatically patches DSH with the needed composition row and admission wrapper.
Free anonymous fallback	An OVHcloud vision endpoint that needs no API key (≈2 req/min per IP). You can also plug in any of the listed free providers (Zhipu, DashScope, Intern AI, etc.) for higher quotas.
No Python	All image processing (down‑scale, grounding, cropping, pixel‑diff, palette, OCR, SVG trace, cut‑out, HTML screenshot) is done with native Node libraries and Chrome.
Multi‑step image workflows	An image turn becomes a normal tool‑calling turn: vision_ground → vision_crop → vision_describe → vision_pixel_diff … allowing the agent to iterate until the task is finished.
Cache by content hash	Vision results are cached; later text turns reuse the stored description instead of re‑calling the vision model.
Stable tool schema	Eleven vision tools are registered at plugin start (or lazily with progressiveTools:true). Tools include vision_describe, vision_ground, vision_crop, vision_pixel_diff, vision_ocr, vision_trace, etc.
Automatic provider fallback	If a provider returns a 402/429/5xx error, the chain silently tries the next provider, finally falling back to the anonymous OVH endpoint.
Transparent UI	Uploaded images stay visible in the chat UI; the routing to vision tools happens only inside the model call, so the conversation log remains clean.

How it works (high‑level)

1. User uploads/pastes an image while using a model group marked “+ Auto Vision”.
2. DSH detects the image turn and rewrites the request so that the vision model is called first, returning a JSON payload (or an annotated PNG) instead of a plain text answer.
3. The LLM (DeepSeek) receives the tool results and can decide to call additional vision tools, cache the result, or continue the conversation.
4. When the vision chain succeeds, the final answer is sent back to the user; if all providers fail, a helpful error message is shown.

Installation & quick start

# one‑liner (recommended)
npx @deepseek-ai/dsh plugin --profile web add dsh-vision-router

# if you run a local deepseek‑harness checkout
cd deepseek-harness
pnpm dsh plugin --profile web add dsh-vision-router

1. Reload/restart DSH Web so the plugin bundle is discovered.
2. In the chat composer open the model selector → choose a model group that now has a “+ Auto Vision” entry (e.g. opencode-go + Auto Vision).
3. Paste or upload an image. The built‑in OVH fallback is already configured, so no API key is required.
4. Use any of the vision tools in the agent’s prompts, e.g.

   vision_ground image="screenshot.png" target="send button"
   vision_crop   image="screenshot.png" region="1067,841,1108,881"
   vision_describe image="crop.png" question="What does this button say?" json=true
   

Free vision back‑ends (you can add them via httpProviders in the plugin config or ~/.dsh/.credentials.yaml):

• OVHcloud anonymous endpoint (2 req/min per IP) – default fallback.
• OVHcloud keyed endpoint (400 req/min per project) – higher quota.
• Zhipu bigmodel.cn (unlimited tokens, China‑direct).
• DashScope, Intern AI, Groq, Google AI Studio, NVIDIA NIM, OpenCode Zen, OpenRouter – each with their own free limits (see the README table).

Typical workflows

• UI verification – vision_html_screenshot → vision_pixel_diff to ensure a rebuilt UI matches a reference image.
• Form automation – vision_ground to locate a button, vision_crop to zoom, vision_ocr to read labels, then send a click via the agent.
• Document extraction – vision_long_screenshot_ocr splits a long screenshot, runs OCR on each chunk, and stitches Markdown output.

Configuration highlights

• progressiveTools: false (default) – all eleven tools are available immediately.
• Set progressiveTools: true in cordis.patch.yml to expose only vision_activate at start and load the rest on demand.
• Vision provider rows can be left empty; the OVH anonymous chain will always be the final fallback.
• Cached results are stored by attachment content hash, enabling “image memory” across turns.

Troubleshooting

• Blank settings page – fixed in v1.4.4; ensure you’re on that version.
• Model‑group selection – you must pick the “+ Auto Vision” entry; otherwise DSH will reject the image before the plugin can act.
• Rate‑limit errors – add a keyed OVH endpoint or another free provider to the chain.

---

Bottom line: dsh‑vision‑router turns image uploads into first‑class tool calls for DeepSeek Harness, preserving pixel fidelity, offering a free anonymous vision fallback, and requiring only a single npm command to get started. It is a genuine, production‑ready AI‑tooling plugin rather than a simple demo or curated list.