firecrawl/pdf-inspector

What it does

pdf‑inspector is a fast, Rust‑based library that looks at a PDF, decides what kind of document it is (plain text, scanned image, mixed, etc.) and then extracts the text in a structured way. It can turn the PDF into clean Markdown, preserving headings, lists, code blocks, tables and even the exact X/Y position of each piece of text. For PDFs that contain images instead of real text, the library can optionally run a lightweight OCR model (PP‑OCRv6 Small) on just the pages that need it, avoiding a full‑document OCR pass.

Why it matters

Most PDFs you encounter are already text‑based, but many pipelines still send every file through expensive OCR services. pdf‑inspector can classify a PDF in 10‑50 ms and extract text in under 200 ms for the majority of files, saving both compute cost and latency. Its position‑aware extraction also makes downstream tasks—like building searchable indexes, feeding LLMs, or converting documents to structured data—much more reliable.

Who would use it

• Developers building document‑processing pipelines (e.g., web‑crawlers, enterprise ingestion services) that need a cheap first‑pass to decide whether OCR is required.
• LLM‑oriented applications that ingest PDFs and want clean Markdown or structured text without paying for external OCR APIs.
• Data‑science teams extracting tables, headings and code snippets from research papers, financial reports, invoices, or legal contracts.
• Front‑end engineers who want to run PDF parsing directly in the browser via WebAssembly.

How it works (high‑level)

1. Fast classification – Samples the PDF’s content streams, looks for text (Tj/TJ) vs. image (Do) operators, and returns a confidence score plus a list of pages that would need OCR.
2. Single‑pass parsing – The file is loaded once and the same in‑memory representation is reused for both classification and extraction, avoiding duplicate I/O.
3. Extraction pipeline – Parses fonts, decodes CID encodings, walks the PDF operators to produce TextItems with font, size and X/Y coordinates, then groups them into lines, detects columns, and determines reading order.
4. Table detection – Uses two strategies: (a) rectangle‑based detection from drawing commands (union‑find on PDF rectangles) and (b) heuristic alignment detection from text positions, handling multi‑page tables and financial layouts.
5. Markdown conversion – Analyzes font size ratios for headings, font names for bold/italic, monospaced fonts for code blocks, list markers, URLs, etc., and emits clean Markdown with optional page‑break markers.
6. Optional OCR – When the ocr feature is enabled, only the pages flagged as non‑text are rasterized and fed to the PP‑OCRv6 Small model, keeping the rest of the fast Rust pipeline untouched.

Getting started (quick steps)

• Rust: cargo add pdf-inspector then call process_pdf("file.pdf").
• Python: Install the maturin build tools, run maturin develop --release, then import pdf_inspector; pdf_inspector.process_pdf("file.pdf").
• Node.js: npm install @firecrawl/pdf-inspector and use processPdf.
• Browser: Install the WASM package @firecrawl/pdf-inspector-wasm, call init() and then processPdf on an Uint8Array of the PDF.
• CLI: cargo install pdf-inspector and run pdf2md file.pdf (or detect-pdf for classification only).

Limitations & open questions

• OCR is optional and requires extra native dependencies (PDFium, ONNX Runtime, the PP‑OCR model). Without the ocr feature the library cannot extract text from pure‑image PDFs.
• Complex layouts (e.g., heavily nested tables, unusual column structures) may still need manual post‑processing; the heuristic table detector works well for most financial and report‑style tables but is not a full‑blown layout engine.
• Language support – The core extraction is language‑agnostic, but OCR quality depends on the PP‑OCR model, which is primarily tuned for Latin scripts and may be less accurate on CJK or handwritten text.
• Performance on huge PDFs – Classification can be tuned with different ScanStrategy options (full scan vs. sampling) to balance speed vs. precision for very large documents.

---

All details are taken directly from the repository’s README.