Aas-ee/open-webSearch

Open‑WebSearch – Live web search for AI agents (no API keys needed)

What it is

• A small Node‑JS service that can be run locally (as an MCP‑compatible server, a CLI tool, or a long‑running HTTP daemon) and lets LLM‑based agents perform live web searches and fetch article content.
• It aggregates results from many public search engines (Bing, DuckDuckGo, Baidu, Brave, Exa, Startpage, Sogou, HackerNews, CSDN, Juejin, etc.) and can retrieve the full text of pages such as GitHub READMEs, CSDN posts, or generic Markdown/HTML pages.
• No external API keys or authentication are required; the only optional requirement is an HTTP proxy if you need to reach restricted sites.

Key features

• Multi‑engine search – pick a default engine or let the user choose; you can limit the allowed engines via ALLOWED_SEARCH_ENGINES.
• Structured output – each result contains a title, URL, and description, ready for LLM consumption.
• Content fetching – dedicated fetch commands for CSDN, GitHub READMEs, Juejin, generic web pages, and (future) other blogs.
• Three ways to run
  1. MCP server – plug into Claude Desktop, Cherry Studio, Cursor, etc.
  2. CLI – one‑off commands like open-websearch search "prompt".
  3. Local daemon – a persistent HTTP service exposing GET /health, POST /search, POST /fetch‑*.
• Proxy support – separate install‑time (npm --proxy …) and runtime (USE_PROXY=true PROXY_URL=…) settings.
• Optional Playwright integration – when installed, the tool can fall back to a headless Chromium browser for sites that block simple HTTP requests (e.g., Bing’s anti‑bot page or JavaScript‑heavy pages). The mode can be forced (SEARCH_MODE=request or playwright) or left to auto‑detect.
• Highly configurable via environment variables – everything from CORS handling to custom tool names for MCP, to persistent browser profile locations.

Typical usage patterns

Scenario	Recommended entry point
Quick one‑off search from a script or terminal	npx open-websearch@latest search "my query" --json
Agent that needs a stable endpoint (e.g., Claude Desktop)	Run open-websearch serve (or npm run serve) and point the agent’s MCP config to http://localhost:3000/mcp
Repeated calls from another program	Start the daemon once (open-websearch serve) and use POST /search or the MCP tool names (search, fetchWebContent, …)
Environments behind a firewall	Set USE_PROXY=true and PROXY_URL=http://127.0.0.1:7890 before starting the daemon

Installation & getting started

# Quick start – no build needed
npx open-websearch@latest               # runs the default mode (STDIO)

# If you want the daemon or MCP server
npm install -g open-websearch          # install globally
npm run build                           # compile TypeScript sources
open-websearch serve                    # start the HTTP daemon (default port 3000)
open-websearch status --json            # check it’s alive

Adding Playwright for better page rendering (optional but recommended for sites that need a real browser):

npm install playwright               # installs Playwright and a bundled Chromium
npx playwright install chromium
SEARCH_MODE=auto npx open-websearch@latest   # daemon will now use the browser when needed

You can also point the service at an existing Chrome/Chromium instance via PLAYWRIGHT_CDP_ENDPOINT or connect to a remote Playwright server with PLAYWRIGHT_WS_ENDPOINT.

Configuration highlights

• DEFAULT_SEARCH_ENGINE – choose the default engine (e.g., duckduckgo).
• ENABLE_CORS – set to true if the daemon will be called from a browser.
• MCP_TOOL_SEARCH_NAME – rename or disable the MCP‑exposed search tool.
• PLAYWRIGHT_HEADLESS – false if you need to see the browser for debugging.
• OPEN_WEBSEARCH_PROFILE_DIR – where persistent browser profiles are stored; delete this folder to clear cookies/cache.

Why it matters for AI agents

• LLMs (Claude, GPT, etc.) are otherwise limited to static knowledge; Open‑WebSearch gives them a live, key‑less bridge to the open internet.
• The MCP interface means the tool plugs directly into existing agent ecosystems without custom code.
• Because it can run entirely locally, you keep data on‑premise and avoid third‑party API costs or rate limits.

Roadmap (as listed in the README)

• Add Google and more mainstream search engines.
• Expand supported blogs, forums, and social platforms.
• Improve article extraction quality and add more site‑specific fetchers.

---

All details above are taken directly from the repository’s README; no additional features have been inferred.