PullMD – Self‑hosted URL‑to‑Markdown service

What it does

PullMD is a self‑hosted web service that turns any public URL (or uploaded file) into clean Markdown. It works for:

• ordinary web pages (using Cloudflare Markdown, Mozilla Readability, Trafilatura, or a headless‑Chromium fallback),
• Reddit and Hacker News threads with full comment trees,
• documents such as PDF, Word, PowerPoint, Excel, EPUB,
• images and audio (captioning / transcription via an OpenAI‑compatible vision or speech model),
• YouTube videos (title, description and transcript). The result is returned as Markdown with a YAML front‑matter block that contains metadata (source URL, fetch date, author, etc.). A short 32‑character share ID can be used to retrieve a cached version that refreshes automatically after an hour.

Who might use it

• AI agents that need a token‑efficient, plain‑text representation of a web page or document.
• Developers building RAG pipelines who want to fetch and normalise content on‑premise.
• Power users who want a personal “read‑later” service that strips ads and boiler‑plate.
• Teams that need a private, controllable alternative to public summarisation services.

How to get started

1. Install Docker‑Compose.
2. Run the one‑liner provided in the README:

   mkdir pullmd && cd pullmd
   curl -O https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/docker-compose.yml
   docker compose up -d
   

3. Open http://localhost:3000 in a browser. No .env file is required; defaults listen on port 3000.
4. (Optional) Add a .env to customise things like authentication, OCR keys, or proxy settings.

Key technical details

• Architecture – The main PullMD container coordinates three optional side‑car services:
  • trafilatura – extracts the main article body from static HTML.
  • playwright – headless Chromium/Firefox/WebKit renderer for JavaScript‑heavy pages.
  • markitdown – converts uploaded documents, images, audio, and YouTube videos to Markdown. These side‑cars are started by default but can be omitted if you only need plain web‑page extraction.
• Output format (v3) – By default the Markdown body contains only the title and content. All other information (URL, fetch date, Reddit stats, OCR/model usage, etc.) lives in the YAML front‑matter, reducing token count for downstream LLM calls. The legacy inline header can be re‑enabled with PULLMD_SOURCE_HEADER=true.
• Security – Built‑in SSRF protection blocks private‑network destinations. Rate‑limits, authentication modes (disabled, single‑admin, multi‑user), and optional sign‑up control are configurable via environment variables.
• Extensibility – Users can add custom site recipes (JSON‑LD to front‑matter, custom selectors) and control which front‑matter fields are emitted with PULLMD_FRONTMATTER_FIELDS.
• Performance – Caching stores results in a SQLite file (/data/cache.db). Cache retention, size limits, and a “coverage guard” that detects overly‑short extractions are all tunable.

When to consider alternatives

• If you only need a quick one‑off conversion, public services like Mercury or browser extensions may be faster.
• For massive, high‑throughput crawling you might prefer a dedicated scraper framework (e.g., Scrapy) combined with a separate Markdown renderer.
• If you require deep semantic summarisation rather than raw Markdown, you’ll need an LLM on top of PullMD’s output.

---

All information above is taken directly from the repository’s README; no external assumptions have been added.