flairNLP/fundus

A very simple news crawler with a funny name

What it solves

Fundus is a news crawler designed to simplify the process of extracting high-quality text and images from online news articles. It removes the noise of web pages to provide clean, structured data for researchers and developers, specifically optimizing for high extraction precision and recall.

How it works

Fundus provides a Python package that allows users to crawl articles from live websites or the CommonCrawl CC-NEWS archive. It uses a PublisherCollection to target specific news sources or regional groups (e.g., US or UK publishers). The crawler can be configured to use multiple CPU cores for performance and can restrict sources to sitemaps for reduced bandwidth usage. It extracts the article title, main body text, URL, and detailed image metadata (including captions and copyright holders).

Who it’s for

Researchers, data scientists, and AI developers who need large-scale, clean news corpora for training AI models or performing linguistic analysis.

Highlights

  • High Extraction Quality: Outperforms other scrapers like Trafilatura and news-please in ROUGE-LSum F1-score benchmarks.
  • Multi-Source Crawling: Supports both live web crawling and the CC-NEWS web archive for massive datasets.
  • Detailed Image Parsing: Extracts not just image URLs, but also captions, descriptions, and multiple resolution versions.
  • Easy Integration: Allows crawling thousands or millions of articles with just a few lines of Python code.

Related

  • Project
    adbar/trafilatura

    A Python package and command-line tool for web crawling and text extraction that turns raw HTML into structured data by removing noise and boilerplate.

  • Project
    oxylabs/AI-Search-py

    A Python-based search agent that allows developers to search the web using natural language and automatically scrape content from the results, including support for JavaScript rendering.

  • Project
    docling-project/docling-parse

    Docling Parse is a fast, configurable PDF‑parsing library (Python API with a C++ core) that extracts text, words, lines, vector shapes and bitmap images together with their coordinates. It supports both sequential and multi‑threaded parsing, lets you choose what to compute or materialize per page, and includes rendering utilities. Ideal for AI workflows that need layout‑aware document data.

  • Project
    0xMassi/webclaw

    A web extraction tool that converts websites into clean Markdown, JSON, and LLM-ready context for AI agents and RAG pipelines.

  • Project
    paulpierre/markdown-crawler

    A multithreaded web crawler that converts website pages into markdown files to simplify document parsing and chunking for RAG and LLM fine-tuning.