Aas-ee/open-webSearch

Open‑WebSearch – a plug‑and‑play web‑search service for AI agents

What it is – Open‑WebSearch is an open‑source tool that runs a local server (or a CLI) capable of querying a variety of public search engines (Bing, DuckDuckGo, Baidu, Brave, etc.) and returning the results in a clean, structured JSON format. It can also fetch the full text of individual pages (GitHub READMEs, CSDN articles, generic Markdown/HTML pages). The service is designed to be used by AI agents (Claude, Cursor, etc.) via the MCP (Model‑Center‑Protocol) interface, but it also works as a simple command‑line utility or a long‑running HTTP daemon.

Why it matters – Modern LLM‑based agents often need up‑to‑date information from the web. Most commercial APIs require paid keys and rate limits. Open‑WebSearch removes that barrier: it scrapes public search engines without any API keys, supports proxy configuration for restricted networks, and can fall back to a headless browser (Playwright) when a site blocks simple HTTP requests. This makes it a practical building block for autonomous agents, RAG pipelines, or any workflow that needs live web lookup.

How it works

• Search – The search endpoint contacts the chosen engine, aggregates titles, URLs, and snippets, and returns them as JSON. You can pick a default engine or limit the allowed list via environment variables.
• Fetch – After a search, you can call fetch‑web (generic pages) or specialized fetchers (fetchGithubReadme, fetchCsdnArticle, etc.) to retrieve the full article body. When Playwright is available, the fetcher can render JavaScript‑heavy pages and reuse cookies from a persistent browser profile.
• Deployment options –
  • MCP server – Connect directly to Claude Desktop, Cherry Studio, VS Code extensions, etc.
  • CLI – One‑off commands like open-websearch search "latest AI news" --json.
  • Local daemon – A continuously running HTTP service (POST /search, POST /fetch‑*) that reduces start‑up overhead for repeated calls.
  • Docker – Official container image (ghcr.io/aas-ee/open-web-search) for quick, isolated deployment.
• Configuration – All behavior is controlled by environment variables (default engine, proxy settings, CORS, Playwright mode, etc.). This makes it easy to adapt to corporate firewalls or to enable/disable the headless‑browser fallback.

Typical use‑case

1. An LLM agent receives a user query like “Find recent papers on diffusion models.”
2. The agent calls the Open‑WebSearch MCP tool search (or the CLI) with the query.
3. The service returns a list of URLs and snippets.
4. The agent picks the most promising result and calls fetchWebContent (or a specialized fetcher) to retrieve the full text.
5. The retrieved content is fed back into the LLM for summarisation or citation.

Key strengths

• No API keys – Works out‑of‑the‑box with public search engines.
• Multi‑engine support – Choose the best engine for a region or language.
• Proxy‑aware – Built‑in HTTP proxy handling for censored or corporate networks.
• Playwright fallback – Handles sites that require JavaScript rendering or cookies.
• Agent‑friendly – Exposes a standard MCP tool interface and a simple HTTP API.

Limitations

• Relies on scraping public search pages, so heavy usage may trigger rate‑limits or captchas.
• Only a subset of engines is currently implemented; Google is not yet supported.
• Browser fallback requires a separate Playwright installation or an existing remote browser.

Bottom line – Open‑WebSearch provides a lightweight, key‑free bridge between LLM agents and the live web, making it a useful component for any RAG or autonomous‑agent stack that needs up‑to‑date information.

---

Quick start

# Run the service via npx (no install needed)
DEFAULT_SEARCH_ENGINE=duckduckgo npx open-websearch@latest

# Or start the daemon for repeated calls
npm install -g open-websearch
open-websearch serve   # then POST /search to http://localhost:3000

---

Further reading – See the README sections Features, CLI and Local Daemon, Installation Guide, and Docker Deployment for detailed commands and environment‑variable options.