alirezamika/autoscraper

What it solves

AutoScraper simplifies web scraping by removing the need to manually write complex CSS selectors or XPath expressions. Instead of hard-coding rules for every website, it allows users to provide a few examples of the data they want to reach, and the tool automatically learns the patterns to find similar elements on the same or other pages.

How it works

The tool takes a URL or HTML content and a list of "wanted" sample data (such as text, URLs, or HTML tag values). It analyzes the page structure to learn scraping rules based on these samples. Once a model is built, it can be used to retrieve either similar results (a list of elements matching the pattern) or exact results (specific elements in the same order as the provided samples).

Who it’s for

Python developers who need to extract data from websites quickly without spending time analyzing HTML structures or maintaining fragile selectors.

Highlights

• Example-based learning: Learn scraping rules automatically from a list of sample data.
• Flexible extraction: Supports retrieving both similar elements (e.g., all post titles) and exact elements (e.g., a specific stock price).
• Model persistence: Ability to save and load learned scraping rules to a file for reuse.
• Custom requests: Supports custom headers and proxies via the requests module.