claw-compactor: a 14-stage fusion pipeline for LLM token compression with reversible retrieval and AST-aware code analysis

What it solves

Claw Compactor addresses the high cost and limited context windows of Large Language Models (LLMs) by reducing the number of tokens required for a prompt without destroying critical structural data. Unlike general-purpose compressors, it prevents the loss of code identifiers, JSON keys, and log patterns that are often deleted by perplexity-based compression methods.

How it works

The project uses a 14-stage "Fusion Pipeline" where text passes through a series of specialized compressors. Each stage is gated; it only runs if the content type (e.g., code, JSON, logs, diffs) matches the stage's purpose.

Key mechanisms include:

• Content-Aware Routing: Auto-detects the language and content type to apply the correct compression logic.
• AST-Aware Analysis: Uses tree-sitter to compress code without shortening identifiers.
• Reversible Compression: Stores original content in a hash-addressed RewindStore, allowing an LLM to retrieve the full original text via a marker ID if needed.
• Immutable Data Flow: Each stage produces a new result rather than mutating data in place, ensuring stability.

Who it’s for

It is designed for developers building AI agents or LLM applications that need to feed large amounts of workspace context, logs, or structured data into a model while minimizing latency and API costs.

Highlights

• Zero Inference Cost: Performs compression without requiring an LLM call.
• High Fidelity: Maintains higher semantic content (ROUGE-L scores) compared to tools like LLMLingua-2.
• Versatile Stages: Includes specialized handlers for base64 images, simhash deduplication, JSON sampling, and git diff folding.
• Extensible: Allows developers to add custom compression stages to the pipeline.