Adaptive PDFs: Embedding Structured Markdown for LLM Extraction
Adaptive PDFs solve the structural loss inherent in the PDF format by embedding a hidden layer of structured Markdown that machine extractors can read, while human viewers see a standard formatted document.
The Problem: PDF as a Visual Format
PDFs are primarily visual instructions for drawing glyphs at specific coordinates rather than structured data. While the "Tagged PDF" specification exists to mark headings and paragraphs, most common export tools—including Chrome's print-to-PDF and many LaTeX configurations—do not produce these tags.
When Large Language Models (LLMs) or text extractors process untagged PDFs, they must reconstruct structure from raw coordinates and font sizes. This often results in broken sentences, flattened tables, and a loss of hierarchy, forcing LLMs to guess where a heading ends and a paragraph begins.
How Adaptive PDFs Work
Adaptive PDFs leverage a property introduced in the PDF 1.4 specification (2001) that allows for the definition of replacement text for marked content.
- Human View: PDF renderers ignore the replacement text and draw the visual content stream as intended.
- Machine View: Text extractors that support this property (such as PyMuPDF and Poppler) return the replacement text instead of the visual text.
By attaching structured Markdown to the content stream via marked-content sequences, the document provides two different outputs from a single .pdf file. A human sees a formatted page, while a machine receives a clean Markdown string containing explicit headers (#), tables, and bullet points.
Performance and Benchmarks
Testing indicates that adding a structured layer has a negligible impact on file size and token counts, while significantly increasing information density for LLMs.
Token and Size Impact
| Document | Pages | Size Change | Normal Tokens | Smart Tokens |
|---|---|---|---|---|
| Resume | 1 | +15.7% | 650 | 668 |
| Textbook | 417 | -8.5%* | 193,064 | 195,858 |
| Novel Chapter | 38 | +4.7% | 16,472 | 15,958 |
| Research paper | 18 | +2.5% | 8,082 | 7,897 |
Note: The size reduction in the textbook was attributed to general PyMuPDF optimization (garbage=3) rather than the adaptive technique itself.
LLM Verification
When uploaded to ChatGPT and Claude, both models returned the embedded Markdown (including # and - symbols) when asked to provide the raw text. This suggests that the models are accessing the embedded structured layer rather than relying solely on layout analysis.
Technical Considerations and Community Feedback
While the approach offers a streamlined way to provide machine-readable data, several technical and security caveats were raised by the community:
Extraction Reliability
Because the technique relies on the extractor honoring the replacement-text property, there is a risk of "silent failure." If a tool ignores the property, it will revert to the messy, untagged visual extraction without notifying the user that a structured version was available.
Security and Prompt Injection
The ability to hide text from humans that is visible to LLMs creates a potential vector for prompt injection. Users could embed malicious instructions or "hidden hints" (e.g., in a resume to influence a hiring AI) that a human reviewer would never see.
Alternative Approaches
Some developers suggest that the industry should move toward HTML or maintain separate Markdown sources. One community member shared a technique for distributing Markdown sources by zipping the PDF with its source files using compression level 0, allowing the file to be read as a PDF by viewers and as a ZIP by those seeking the source.
Implementation
The project is open source and available on GitHub at github.com/iminoaru/adaptivepdf. The author is currently exploring a Google Docs extension to simplify the creation of these adaptive documents.