Pulpie: Pareto-Optimal Models for Cleaning the Web

Pulpie: Pareto-Optimal Models for Cleaning the Web

Pulpie is a family of Pareto-optimal models designed to extract main content from HTML pages by removing boilerplate such as navigation, ads, and footers. By utilizing an encoder architecture instead of a decoder, Pulpie achieves state-of-the-art (SOTA) extraction quality while reducing costs by up to 20x compared to leading extractors like Dripper.

High-Impact Data Cleaning for LLM Training and Inference

Clean data is critical for both the pre-training and inference phases of large language models (LLMs). Research indicates that model-based extraction significantly outperforms heuristic-based methods in preserving structured content like code blocks and formulas, where heuristics often corrupt the data.

According to AICC (Ma et al., 2025), improving the extractor alone can lead to measurable gains in model accuracy. A model trained on a model-extracted corpus scored 1.08 percentage points higher in average accuracy across 13 benchmarks, beating models trained on heavily filtered corpora like FineWeb and RefinedWeb.

At inference time, noise in the context window can derail model answers. A single irrelevant passage can reduce accuracy and efficiency, making high-quality cleaning essential for RAG pipelines.

Architectural Shift: From Bandwidth-Bound to Compute-Bound

Pulpie's efficiency gains come from its architecture. While reading extractors like Dripper use a decoder that emits labels one token at a time—making them bandwidth-bound and expensive—Pulpie uses an encoder that labels every HTML block in a single forward pass. This shifts the bottleneck from memory bandwidth to compute, which is significantly more efficient on modern GPUs.

This architectural difference is particularly evident on cheaper GPUs like the NVIDIA L4. On an L4, pulpie-orange-small processes 13.7 pages per second, compared to Dripper's 0.68 pages per second. This results in a massive cost difference when scaling to a billion pages:

Setup Pages/sec (L4) Cost / 1B pages (L4)
Pulpie Small 13.7 ~$7,900
Dripper 0.68 ~$159,000

Model Family and Performance Benchmarks

Pulpie consists of a teacher model and two distilled students. All models are built on EuroBERT and use a <|sep|> block-marker architecture.

Model Specifications

Model Parameters ROUGE-5 F1 Notes
Pulpie Orange Large 2.1B 0.873 Teacher model
Pulpie Orange Base 610M 0.863 Distilled from Large
Pulpie Orange Small 210M 0.862 Recommended for production

Quality Comparison

On the WebMainBench English subset, Pulpie Orange Small matches Dripper's quality (0.862 vs 0.864 ROUGE-5 F1) while being a third of the size. Pulpie also handles long pages better than Dripper; because Pulpie packs blocks into 8,192-token chunks, it does not suffer from the 32k-token context window failures that cause Dripper to return empty results on 135 pages.

Training and Distillation Pipeline

The Pulpie models were trained using a a high-quality synthetic dataset created from 16,670 English Common Crawl pages. Each page was split into blocks using MinerU-HTML and labeled as content or content-boilerplate by DeepSeek V3.2. To ensure label quality, labels were cross-validated with Dripper 0.6B, keeping only pages where the two labelers agreed on at least 70% of thees blocks.

  1. Teacher Training: A EuroBERT-2.1B model was fine-tuned on the validated dataset using class-weighted cross-entropy to handle the imbalance (content rate of 28.6%).
  2. Distillation: The 2.1B teacher was distilled into the Base (610M) and Small (210M) models using a combination of KL-divergence loss (weighted 0.7) and hard-label cross-entropy (0.3) at a temperature of 2.0.

Implementation and Usage

Pulpie is available as a Python package and open-sourced on Hugging Face. Users can choose the model size based on their needs for speed versus quality.

from pulpie import Extractor

# Defaults to Pulpie Orange Small
extractor = Extractor()
result = extractor.extract(html)

print(result.markdown) # Clean markdown output

For bulk processing, the Pipeline class allows overlapping CPU preprocessing with GPU inference:

from pulpie import Extractor, Pipeline, PageInput

pipeline = Pipeline(model="small")
results = pipeline.extract_batch(
    [PageInput(html=h, page_id=i) for i, h in enumerate(pages)]
)

Community Feedback

While the release has received praise for its architectural insight, some community members have questioned the necessity of model-based extraction over simple CSS selectors or markdown converters. However, the data provided by the author shows that heuristic-based tools like Trafilatura score significantly lower on ROUGE-5 F1 (0.619) compared to Pulpie Orange Small (0.862), illustrating the gap in quality for web-scale cleaning.

Sources