coder-hxl/x-crawl

What it solves

Traditional web crawlers often break when websites update their HTML structure or class names, as they rely on fixed selectors to find data. x-crawl solves this by integrating AI to understand the semantic meaning of a page, allowing it to extract information accurately even when the underlying code changes.

How it works

The library combines a standard Node.js crawler with AI integrations (via OpenAI and Ollama). It can crawl static and dynamic pages, handle interface data, and download files. When the AI is used, the developer can pass HTML content to the AI with natural language instructions (e.g., "Get the image links and remove duplicates") to parse and extract specific elements without needing hard-coded CSS selectors.

Who it’s for

Developers who need a robust web scraping tool that can adapt to website changes and those who want to automate complex browser interactions and data extraction using LLMs.

Highlights

• AI-Powered Parsing: Uses LLMs to parse semantic information, reducing the need for manual selector updates.
• Dynamic Page Support: Supports automated operations, keyboard input, and event handling on dynamic pages.
• ** uma Fingerprinting**: Includes device fingerprinting to avoid detection and tracking.
• Advanced Crawling Controls: Features proxy rotation, priority queues, customizable retry logic, and flexible interval timing (fixed or random).
• TypeScript Support: Fully implemented with generics for strong typing.