adbar/trafilatura

What it solves

Trafilatura is designed to gather text from the web and convert raw HTML into structured, meaningful data. It solves the problem of "noise" in web pages—such as headers, footers, and boilerplate—by focusing on the actual main content and metadata, providing a robust and fast way to extract essential information without requiring a database.

How it works

It functions as a Python package and command-line tool that combines web crawling, downloading, and scraping. It uses a rule-based extractor (with fallbacks to jusText and readability-lxml) to identify the main text and metadata (like author, date, and title). It supports various input sources, including live URLs, sitemaps, feeds (RSS, ATOM, JSON), and previously downloaded HTML files, and can output the results in formats like JSON, Markdown, TXT, CSV, and XML.

Who it’s for

It is primarily for developers, researchers, and data scientists who need to create text databases or corpora for NLP tasks, as well as anyone needing to clean web content for structured data extraction.

Highlights

• Comprehensive Pipeline: Combines discovery (sitemaps/feeds), downloading, and extraction in one tool.
• High Performance: Consistently outperforms other open-source libraries in text extraction benchmarks.
• High Precision: Effectively removes boilerplate and noise to isolate the main content.
• Modular and Handy: No database required; supports multiple output formats including XML-TEI.
• Wide Adoption: Used by organizations like HuggingFace, NVIDIA, and Microsoft Research.