coder-hxl/x-crawl
Flexible Node.js crawler library
What it solves
Traditional web crawlers often break when websites update their HTML structure or class names, as they rely on fixed selectors to find data. x-crawl solves this by integrating AI to parse the semantic meaning of a page, allowing it to extract information accurately even when the underlying code changes.
How it works
The library combines a standard Node.js crawler with AI integrations (via OpenAI and Ollama). It can be used as a traditional crawler for navigating pages and interacting with elements, or it can pass HTML content to an AI model to identify and extract specific data points based on natural language instructions rather than rigid CSS selectors.
Who it’s for
Developers who need to build resilient web scrapers that can handle dynamic content and frequent website layout changes without constant manual maintenance.
Highlights
- AI-Powered Extraction: Uses LLMs to understand page semantics and extract data, reducing reliance on fixed class names.
- Comprehensive Crawling: Supports static pages, dynamic pages, interface data, and file downloads.
- Browser Automation: Includes capabilities for keyboard input, event operations, and automated page control.
- Anti-Detection: Features device fingerprinting and rotating proxies to avoid being tracked or blocked.
- Flexible Execution: Supports both synchronous and asynchronous modes, priority queues, and customizable retry logic.
Related
- Project
- Project
- Project
- Project
- Project