fivetaku/insane-search

📚 What insane‑search is

insane‑search is a Claude Code plugin that lets the Claude Code LLM read any public web page, even when the site returns a 403, a CAPTCHA, a WAF block, or otherwise hides its content from ordinary HTTP fetches. It does this without requiring any API keys or proxy configuration – the plugin automatically tries a cascade of public‑access routes (feeds, public APIs, TLS‑impersonated curl calls, and finally a headless browser) until it can retrieve the raw text, metadata (OGP, JSON‑LD) or YouTube captions.

The repository is a genuine software project aimed at extending the capabilities of an AI agent (Claude Code) for web‑retrieval, summarisation and RAG‑style data gathering.

---

⚡ How to get it

/plugin marketplace add https://github.com/fivetaku/gptaku_plugins.git
/plugin install insane-search@gptaku-plugins
/reload-plugins

After the three commands the plugin is ready – you don’t need to learn any new CLI; just ask Claude as you normally would and the plugin will activate automatically whenever a fetch is blocked.

---

🚀 Key capabilities

Feature	What it does
Zero‑setup	Installs required helpers (curl_cffi, yt‑dlp, PDF parsers, etc.) on first use.
Multi‑phase escalation	Tries public endpoints → lightweight probes → TLS‑impersonated curl → real headless browser, stopping as soon as one succeeds.
Human‑like browsing	Uses full TLS fingerprints, cookie warming and referer chains so the request looks like a real browser.
Hidden‑API discovery	Captures a page’s own network calls and re‑uses the JSON the site returns internally.
Broad platform support	Reddit, X (Twitter), YouTube, Hacker News, Naver, Coupang, LinkedIn, Medium, Substack, arXiv, GitHub, Stack Overflow, Bluesky, Mastodon, and any site with a public page, feed or /rss.
PDF & YouTube handling	Lazy‑loaded PDF parsers (pdfplumber, pypdf) and YouTube caption extraction via yt‑dlp.
Structured output	--json-content returns the page body together with metadata and a diagnostic trace in a single request.
Respect for auth walls	Detects login or paywall pages and reports "authentication required" instead of trying to bypass them.

---

🛠️ Typical workflow

1. Ask Claude a normal question that requires web data, e.g.
   • “Find what people are saying about Claude Code on Reddit and summarise the top threads.”
   • “Summarise this YouTube video.”
2. Claude attempts to fetch the URL. If the fetch is blocked, insane‑search automatically runs its escalation pipeline.
3. The plugin returns the extracted text (or captions, PDF text, etc.) together with any discovered metadata.
4. Claude uses that content to answer your original query.

---

⛔ Boundaries & responsible use

• Public‑only: the tool stops at login screens or paywalls; it never logs in or stores credentials.
• Legal compliance: users must respect each site’s Terms of Service, robots.txt, rate limits and local law.
• Dual‑use disclaimer: the repository includes a DISCLAIMER.md that outlines acceptable‑use policies.

---

📄 License

MIT – see the repository’s LICENSE and DISCLAIMER.md for the full terms.

---

TL;DR

insane‑search is a Claude Code plugin that makes the LLM capable of reading any publicly accessible web content by automatically trying a series of increasingly sophisticated fetch methods, all without API keys or manual configuration. It is a practical tool for building RAG pipelines or any AI‑assistant workflow that needs reliable, up‑to‑date web information.