AeternaLabsHQ/pullmd

PullMD – Self‑hosted URL‑to‑Markdown service

What it is

• A self‑hosted web service that fetches a web page (or many other media types) and returns a clean, token‑efficient Markdown document.
• Designed for both humans and AI agents that need a concise, structured representation of online content.

Key capabilities

Capability	Details
Web page extraction	Uses a cascade: Cloudflare native Markdown → Mozilla Readability → Trafilatura → (fallback) headless Chromium via Playwright. Handles Reddit and Hacker News threads with full comment trees.
Document conversion (v3)	PDFs, Word/PowerPoint/Excel, EPUB, etc. can be uploaded or supplied by URL. Optional OCR tier (?pdf=ocr) produces high‑quality tables.
Media handling	Images → captioning; audio → transcription; YouTube URLs → title, description, and timestamped transcript. All powered by any OpenAI‑compatible vision/STT endpoint you configure.
Output format	Markdown body contains only # Title + content. All metadata (source URL, fetch date, author, subreddit, up‑votes, OCR/model usage, etc.) lives in a YAML front‑matter block, which can be trimmed with PULLMD_FRONTMATTER_FIELDS.
Shareable IDs	Every conversion gets an 8‑hex ID. GET /s/:id returns the cached Markdown and automatically refreshes it if older than one hour.
API surface	• GET /api?url=… – one‑shot conversion (web only). • POST /api/file – upload a document, image, or audio for conversion. • POST /mcp – MCP (Claude‑compatible) endpoint for stateless streaming.
Authentication	Three modes: disabled (open), single-admin (one admin user), multi-user (self‑service sign‑up). Admins can manage users via the provided CLI.
Security	Built‑in SSRF protection blocks private, loopback, link‑local and cloud‑metadata addresses. Hosts can be whitelisted with PULLMD_ALLOWED_HOSTS.
Extensibility	Site‑specific “recipes” (JSON‑LD to front‑matter, custom selectors) can be added via PULLMD_SITE_RECIPES.
Monitoring	Side‑car health endpoint (GET /api/status) reports 503 if a renderer (Playwright, MarkItDown, etc.) is down.

How to run

1. Docker Compose (recommended) – pull the ready‑made multi‑arch images from Docker Hub and start with a single command:

   mkdir pullmd && cd pullmd
   curl -O https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/docker-compose.yml
   docker compose up -d   # service listens on http://localhost:3000
   

   No .env file is required; sensible defaults are baked in. Override any setting by creating a .env next to the compose file.
2. Optional sidecars – The stack includes three auxiliary containers:
   • pullmd-trafilatura – Trafilatura extraction service.
   • pullmd-playwright – Headless Chromium/Firefox/WebKit renderer for JS‑heavy pages (adds ~3.7 GB to the image cache). Omit if you only need static extraction.
   • pullmd-markitdown – Document‑to‑Markdown converter (required for PDF/Office/EPUB). Omit to disable document conversion.
3. Local development – Clone the repo, run npm install, then npm start.

Configuration highlights (set in a .env file)

• PUBLIC_URL – Public origin used in help pages and the downloadable Claude skill.
• PULLMD_AUTH_MODE – disabled / single-admin / multi-user.
• PULLMD_ADMIN_EMAIL / PULLMD_ADMIN_PASSWORD – bootstrap admin credentials.
• PULLMD_VISION_API_KEY, PULLMD_STT_API_KEY – enable image captioning or audio transcription via any OpenAI‑compatible endpoint.
• PULLMD_PDF_OCR_API_KEY – turn on the premium OCR tier for PDFs (?pdf=ocr).
• PULLMD_FRONTMATTER_FIELDS – whitelist front‑matter keys to keep token usage low for LLM pipelines.
• DISABLE_PUBLIC_HISTORY – hide the global conversion history for privacy‑focused deployments.
• PULLMD_ALLOWED_HOSTS – whitelist internal hosts if you need to fetch them despite the default SSRF block.

Why it matters for AI/LLM workflows

• Token efficiency: By moving all metadata to front‑matter and emitting a minimal body, the Markdown can be fed to LLMs without wasting context on repeated URLs or timestamps.
• Agent‑ready: The MCP endpoint lets Claude‑style agents request conversions via a streaming‑compatible HTTP call.
• Self‑hosted control: No third‑party API keys are required for basic web extraction, keeping data private and costs predictable.
• Extensible media handling: Images and audio can be turned into captions/transcripts on‑the‑fly, useful for multimodal agents.

Typical usage

# One‑off web page to markdown
curl "http://localhost:3000/api?url=https://example.com"

# Upload a PDF for conversion (requires the markitdown sidecar)
curl -F file=@report.pdf http://localhost:3000/api/file

# Get a shareable markdown via ID
curl http://localhost:3000/s/1a2b3c4d

License – AGPL‑3.0 (see LICENSE).

---

All details above are taken directly from the repository’s README; no additional features have been inferred.