Anakin-Inc/anakin

What it solves

AnakinScraper is an open-source web scraping API designed specifically for AI applications. It solves the problem of converting complex, modern websites into clean, LLM-ready markdown or structured JSON data, while bypassing anti-bot protections and reducing the cost of using paid scraping services.

How it works

The system uses a "handler chain" to maximize efficiency and minimize cost. It attempts to scrape a page using the fastest method first (HTTP fetch), then falls back to a more powerful anti-detect browser (Camoufox/Firefox), and finally to an optional external API handler if local methods fail.

Key technical components include:

• Anti-detect Browser: Uses Camoufox to provide realistic browser fingerprints to avoid detection.
• Proxy Auto-selection: Employs Thompson Sampling (a machine learning technique) to automatically pick the most successful proxy for a specific domain in real-time.
• AI Extraction: Integrates with Google Gemini to transform raw page content into structured JSON.
• Conversion: Automatically removes boilerplate to produce clean markdown for RAG pipelines.

Who it’s for

It is built for developers creating AI agents, RAG (Retrieval-Augmented Generation) pipelines, and data extraction tools that require high-quality, clean web data without managing complex scraping infrastructure.

Highlights

• Cost-Efficient Fallback: Only calls paid APIs for the small percentage of sites that local handlers cannot resolve.
• Zero-Config Start: Can be run as a single Go binary without a database for immediate use.
• ML-Powered Proxies: Real-time learning of the best proxy per domain via Thompson Sampling.
• LLM-Ready Output: Direct conversion to markdown and structured JSON via Gemini AI.
• Full Management UI: Includes a React-based dashboard for tracking jobs, monitoring proxy scores, and managing domain configurations.