sammcj/ingest

What it solves

Ingest simplifies the process of providing large amounts of context (like entire codebases or web pages) to LLMs. It solves the problem of manually copying and pasting multiple files or URLs, by aggregating them into a single, AI-ready Markdown file with a clear directory structure.

How it works

The tool parses directories of plain text files, source code, or web URLs and converts them into a formatted Markdown document. It includes several features to optimize the context window:

• Aggregation: Combines multiple files and directories into one output.
• Code Compression: Uses Tree-sitter to extract only the structural elements (signatures, imports, declarations) of code, removing implementation details to save tokens.
• Web Crawling: Converts web pages and PDFs to Markdown for ingestion.
• Token Counting: Provides offline token estimates (with correction factors for different models) or exact counts via the Anthropic API.
• LLM Integration: Can send the final prompt directly to any OpenAI-compatible API.

Who it’s for

Developers and AI power users who need to feed entire projects, documentation sites, or complex file structures into an LLM for analysis, refactoring, or explanation.

Highlights

• Tree-sitter Compression: Reduces token usage by stripping code bodies while keeping the architecture.
• Flexible Filtering: Use glob patterns to include or exclude specific files.
• Integrated Tokenizer: Bundled offline tokenizers (o200k_base, cl100k_base) for fast, local token counting.
• Web-to-Markdown: Built-in crawling and PDF conversion for external documentation.
• Git Integration: Ability to include git diffs and logs in the prompt.