Episkey-G/GrokSearch-rs

GrokSearch‑rs – a lightweight Rust MCP server for web‑search tools

What it is – A small, self‑hostable Rust binary that implements the MCP (Model‑Client‑Protocol) server used by Claude‑style assistants. It offers a set of tools (web_search, get_sources, web_fetch, web_map, doctor, optional x_search) that let an LLM retrieve, cite, and render live web content. The server can run locally via stdio (launched by the client) or be exposed as a streamable HTTP endpoint for mobile or multi‑device use.

Why it matters – Modern LLM agents need up‑to‑date, verifiable information. GrokSearch‑rs provides that by:

• Querying several external search providers (Tavily, Exa, TinyFish, Firecrawl) in a configurable order.
• Enriching results with specialist extractors for GitHub, StackExchange, arXiv, and Wikipedia that work without any API key.
• Keeping a session cache so follow‑up calls (get_sources) can retrieve the same citations without re‑searching.
• Respecting token limits: it trims the response to stay under a configurable character budget and lets you choose a concise or detailed payload.
• Supporting both xAI’s Grok Responses API and any OpenAI‑compatible chat‑completion gateway, selectable via environment variables.

Key features (as listed in the README)

Feature	What it does
Live web search with citations	Returns a summary plus metadata for the top sources; include_content can embed full extracted text.
Response budgeting	Limits inline source text (max_inline_sources) and total characters (response_max_chars).
Structured web_fetch	Parses GitHub issues/PRs, StackExchange threads, arXiv papers, Wikipedia pages into clean Markdown; falls back to the generic source chain for other URLs.
Two transports	Native xAI Responses (/v1/responses) or any OpenAI‑compatible chat/completions endpoint.
Optional Grok OAuth mode	Stores a local OAuth token so the server can run without an API key.
Remote HTTP mode (feature http)	Multi‑tenant, streamable HTTP server; keys are sent per‑request via headers, never stored on the server.
Pluggable source chain	Ordered providers (Tavily → Exa → TinyFish → Firecrawl) with round‑robin key rotation and fail‑over.
X/Twitter search (x_search)	Available only on the Grok Responses transport.
Concurrency	Handles up to 8 simultaneous stdio tool calls; extra calls are queued.
doctor tool	Quick health‑check that reports connectivity and redacted configuration.

Installation

npm install -g grok-search-rs   # pulls a pre‑built native Rust binary

The installed grok-search-rs command is what an MCP client launches.

Running options

• StdIO (local) – default. The client starts the binary as a subprocess and communicates over stdin/stdout.
• Remote HTTP – compile with --features http (or use the provided Docker image). The server then listens on a configurable address (GROK_MCP_BIND) and serves the same tools over a streamable HTTP endpoint.

Configuration – All settings are supplied either as environment variables (stdio mode) or as HTTP headers (remote mode). A single global TOML file (~/.config/grok-search-rs/config.toml) can hold the keys so multiple clients share them.

Core variables (excerpt):

GROK_SEARCH_API_KEY   – Bearer token for xAI/Grok (or leave empty for OAuth)
GROK_SEARCH_URL       – Base URL of the Grok gateway (default https://api.x.ai)
TAVILY_API_KEY        – Required for generic web search; can be a comma‑separated list for rotation
FIRECRAWL_API_KEY     – Enables Firecrawl fallback
EXA_API_KEY, TINYFISH_API_KEY – Enable those providers in the chain
GROK_SEARCH_MODEL     – Model name to send (e.g. grok-4.20-fast)
GROK_SEARCH_RESPONSE_MAX_CHARS – Whole‑response budget (default 45 000)

Headers for the HTTP transport follow the same naming scheme, e.g. X-Grok-Api-Key, X-Tavily-Api-Key.

Typical workflow

1. Search – Call web_search with a query. The server returns a concise answer plus a list of sources; the top max_inline_sources may include extracted text.
2. Fetch details – Use web_fetch on any source URL to get clean Markdown (specialist extractors give richer output for GitHub, StackExchange, arXiv, Wikipedia).
3. Paginate sources – If the original search yielded many citations, get_sources lets the client page through them.
4. Diagnostics – Run doctor to verify that all upstream APIs are reachable and that the config was loaded correctly.

Self‑hosting – Pre‑built binaries for Linux x86_64 and aarch64 are released, as is a Docker image (ghcr.io/episkey-g/grok-search-rs). The Dockerfile and accompanying docker‑compose.yml/Caddyfile enable a one‑command HTTPS deployment.

Development – The repo is a standard Rust project. Building from source:

cargo build --release               # stdio binary
cargo build --profile release-http --features http   # HTTP‑enabled binary

Standard Rust tooling (cargo fmt, cargo clippy, cargo test) is used.

Who might use it – Anyone building an LLM‑powered assistant (Claude, ChatGPT, custom agents) that needs reliable, cited web search capabilities, especially when they want full control over keys, caching, and response size.