itsOwen/CyberScraper-2077

What it solves

CyberScraper 2077 is an AI-powered web scraping tool designed to extract structured data from websites, including those with anti-bot defenses and .onion sites on the Tor network. It simplifies the process of turning unstructured web content into usable formats like JSON, CSV, and Excel.

How it works

The tool uses a combination of Playwright for browser automation and Large Language Models (LLMs) from OpenAI, Gemini, or local models via Ollama and LiteLLM to intelligently parse and structure web content. It includes stealth parameters to avoid detection, supports asynchronous operations for speed, and can route traffic through the Tor network for dark web scraping. It also features a Streamlit-based GUI for user interaction.

Who it’s for

It is intended for data analysts, researchers, security analysts, and investigators who need to extract data from the web anonymously and efficiently.

Highlights

• AI-Driven Parsing: Uses LLMs to understand and parse web content intelligently.
• Tor Network Support: Ability to scrape .onion sites with automatic routing.
• Stealth Capabilities: Includes features to bypass bot detection and captchas.
• Multi-Format Export: Supports exporting data to JSON, CSV, HTML, SQL, Excel, and Google Sheets.
• Multi-Page Scraping: Beta support for scraping paginated content using custom URL patterns.
• Local LLM Integration: Compatible with Ollama for running open-source models locally.