emcf/thepipe

thepipe – AI‑enhanced document extraction

What it does
thepipe (installed via the thepipe-api package) turns messy files—PDFs, Word docs, PowerPoints, notebooks, audio/video, images, ZIPs, even whole directories—into clean markdown, tables, and images. It can also run OCR on image‑only scans and pull out structured data that matches a user‑provided Pydantic schema.

How it works

• The core scrape_file function reads a file, optionally sends page images to a vision‑language model (VLM) such as an OpenAI multimodal model, and returns a list of chunks (objects containing text, optional images, and metadata).
• Chunking can be done by page, length, markdown section, keywords, or—if you install the optional sentence‑transformers extra—by semantic change. An experimental agentic chunker uses an LLM to decide splits.
• For audio/video, the optional openai‑whisper extra transcribes the media and extracts representative frames.
• Extracted chunks can be turned into OpenAI‑style chat messages (chunks_to_messages) or converted to LlamaIndex Document objects (Chunk.to_llamaindex).
• Structured extraction leverages OpenAI’s structured output feature: you pass a Pydantic model and the library returns validated instances.

Key features

• Multimodal AI scraping – works with any OpenAI‑compatible client (OpenAI, OpenRouter, local OpenLLM, etc.).
• Pluggable extras – lightweight default install; add [audio], [semantic], [llama-index], or [gpu] for Whisper, sentence‑transformers, LlamaIndex support, or full GPU acceleration.
• CLI – thepipe <source> with flags for API keys, model selection, text‑only output, inclusion regex, and verbosity.
• Integration‑ready – output can be fed directly to LLMs, stored in vector DBs (ChromaDB, LlamaIndex, etc.), or used in RAG pipelines.

Typical usage

from thepipe.scraper import scrape_file
from openai import OpenAI

client = OpenAI()                     # any OpenAI‑compatible client
chunks = scrape_file(
    "paper.pdf",
    openai_client=client,
    model="gpt-5.6-luna",
    include_input_images=True,
)

# Send to an LLM as a chat prompt
from thepipe.core import chunks_to_messages
messages = [{"role": "user", "content": [{"type": "text", "text": "Summarize this paper."}]}]
messages += chunks_to_messages(chunks)
response = client.chat.completions.create(model="gpt-5.6-luna", messages=messages)

Installation

pip install thepipe-api               # CPU‑only base
pip install thepipe-api[audio]        # Whisper for audio/video
pip install thepipe-api[semantic]     # sentence‑transformers for semantic chunking
pip install thepipe-api[llama-index]  # LlamaIndex conversion support
pip install thepipe-api[gpu]          # PyTorch + full GPU stack

(For GPU‑heavy extras you may need to install the CPU wheels from the PyTorch index first.)

When to use it

• You need clean, markdown‑ready text from PDFs, slides, or scanned images.
• Your downstream pipeline relies on multimodal LLMs or vector stores.
• You want a single library that can also handle audio/video transcription and structured data extraction without stitching together many separate tools.

Limitations

• VLM‑based scraping requires an OpenAI‑compatible API key (or a self‑hosted VLM endpoint).
• GPU acceleration is optional; CPU‑only runs will be slower for vision‑heavy workloads.
• Some features (semantic/agentic chunking, structured extraction) need extra dependencies.

---

The information above is taken directly from the project's README.