D4Vinci/Scrapling

What it solves

Scrapling is an adaptive web scraping framework designed to simplify data extraction from the modern web. It addresses common scraping challenges such as website design changes that break selectors, anti-bot systems (like Cloudflare Turnstile), and the complexity of scaling from a single request to a full-scale concurrent crawl.

How it works

The framework provides a suite of tools for different scraping needs:

• Adaptive Parsing: The parser can learn from website changes and automatically relocate elements when page structures update.
• Stealthy Fetching: It includes fetchers that bypass anti-bot protections and support headless browser automation.
• Spider Framework: A Scrapy-like API allows users to define spiders with async callbacks, concurrent crawling, and automatic proxy rotation.
• AutoThrottle: The system dynamically adjusts crawl speeds based on website response times and backing off when rate-limiting is detected.
• Persistence: It supports checkpoint-based pause and resume functionality to ensure long-running crawls are not lost.

Who it’s for

It is built for web scrapers and developers who need a reliable, scalable way to collect data from websites, ranging from simple scripts to enterprise-grade crawls.

Highlights

• Adaptive Selectors: Automatically finds elements even after website design updates.
• Anti-Bot Bypass: Out-of-the-box handling of Cloudflare Turnstile and stealthy fetchers.
• Scalable Crawling: Supports concurrent, multi-session crawls with real-time streaming of results.
• Developer-Friendly: Includes a development mode to cache responses and replay them to iterate on parsing logic without re-hitting servers.
• Specialized Templates: Ready-made spiders for Shopify stores, sitemaps, and XML/RSS feeds.