itsOwen/CyberScraper-2077

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

What it solves

CyberScraper 2077 automates the extraction of structured data from websites. It addresses the complexity of modern web scraping by using LLMs to parse and format unstructured web content into usable formats like JSON, CSV, or SQL, while providing tools to bypass common anti-bot protections.

How it works

The project uses a Streamlit-based interface to interact with a backend powered by Playwright for browser automation. Users provide a URL, and the tool uses LLMs (via OpenAI, Gemini, or local models via Ollama/LiteLLM) to intelligently interpret and extract specific data points. It includes features for multi-page navigation, Tor network routing for anonymity, and stealth parameters to mimic human browsing behavior.

Who it’s for

This tool is designed for data analysts, researchers, and developers who need to collect data from websites efficiently. It is particularly useful for those dealing with paginated content or sites that require advanced techniques to avoid detection.

Highlights

  • AI-Powered Parsing: Leverages LLMs to understand and structure web content automatically.
  • Stealth & Anti-Bot: Includes features like Playwright stealth mode, custom headers, and local browser instance usage to bypass bot detection.
  • Multi-Format Export: Supports saving extracted data to JSON, CSV, HTML, SQL, Excel, or directly to Google Sheets.
  • Tor Integration: Allows for anonymous scraping of .onion sites.
  • Flexible Model Support: Compatible with cloud-based APIs (OpenAI, Gemini) and local open-source models via Ollama.

Related

  • Project
    ScrapeGraphAI/Scrapegraph-ai

    An AI-powered web scraping library that uses LLMs and graph logic to extract structured data from websites and local documents using natural language prompts.

  • Project
    Anakin-Inc/anakin

    An open-source web scraping API that converts websites into LLM-ready markdown or structured JSON, featuring an anti-detect browser and ML-powered proxy selection.

  • Project
    oxylabs/ai-crawler-py

    An AI-powered web crawler that uses natural language prompts to discover relevant pages and extract structured JSON or Markdown data from websites.

  • Project
    mishushakov/llm-scraper

    LLM Scraper is a TypeScript library that lets you extract structured data from any webpage using large language models. It combines browser automation with schema-based prompting to turn raw web content into organized JSON, supporting multiple LLM providers and streaming output.

  • Project
    coder-hxl/x-crawl

    A Node.js crawler library that integrates with AI models like OpenAI and Ollama to extract data from websites semantically, making scrapers more resilient to layout and structural changes.