ai-crawler-py: a low-code AI web crawler that extracts structured data using natural language prompts and automated schemas

What it solves

It eliminates the need to build and maintain custom web scrapers with static CSS or XPath selectors. Instead of writing complex scripts to find specific data on a website, users can describe what they need in plain English, and the tool handles the discovery and extraction of that information.

How it works

Users provide a starting URL and a natural language prompt describing the desired content. The AI agent then intelligently explores the domain, identifies relevant pages, and extracts the data. The output can be delivered as Markdown or as structured JSON; for the latter, users can either provide an OpenAPI schema or have the AI generate one from a prompt to ensure the data fits their application's requirements.

Who it’s for

It is designed for developers and data scientists who need to acquire web data for analysis or automation pipelines without spending time on manual scraper development.

Highlights

• Natural Language Control: Use plain English prompts to guide the crawl agent and define data needs.
• AI-Driven Discovery: Automatically identifies and prioritizes pages most aligned with the user's prompt.
• Flexible Output: Supports both Markdown and structured JSON formats.
• Automated Schema Generation: Can automatically create parsing schemas from natural language descriptions.
• Technical Versatility: Handles both static and JavaScript-rendered pages with optional geo-location targeting.