jia-gao/leanctx

Drop-in prompt compression for production LLM apps. Cut your token bill 40-60% without changing your code. Python SDK, LLMLingua-2, MIT.

What it solves

leanctx is a drop-in prompt compression library designed to reduce LLM input-token costs (by 10–40%) without significantly sacrificing accuracy. It specifically targets dynamic per-query content—such as chat history, retrieved documents in RAG, and tool outputs—which cannot be handled by provider-side prompt caching.

How it works

The project uses a "loss-tolerance routing" mechanism to classify prompt segments by how much distortion they can survive. It then applies different compression strategies based on the content type:

  • Zero tolerance (Verbatim): Code, stack traces, and tool-call metadata are kept byte-for-byte to ensure structural integrity.
  • High tolerance (Lingua): Documentation, logs, and retrieved passages are compressed using a local LLMLingua-2 model (~50% reduction).
  • Conditional (Self-LLM): Low-confidence prose or oversized context is routed to a cheap LLM for summarization (opt-in).

To ensure reliability, the library checks "invariants" (e.g., message order, tool IDs) after compression. If a check fails or the process times out, it "fails open" by sending the original uncompressed request to the provider.

Who it’s for

  • Developers of production LLM applications with high token bills.
  • Builders of RAG applications with large retrieved contexts.
  • Creators of long-running conversational agents (e.g., using LangChain or CrewAI).
  • Developers of coding agents that handle large tool-call histories and source code.

Highlights

  • Drop-in Integration: Wraps existing OpenAI, Anthropic, and Gemini SDKs with a simple configuration change.
  • Local Execution: Runs compression models locally (MIT-licensed), ensuring user data does not leave the infrastructure by default.
  • Complementary to Caching: Works alongside provider prompt caching by compressing the variable suffix of a request.
  • Observability: Built-in OpenTelemetry support for tracking token savings, costs, and latency.
  • HTTP Sidecar: Includes a FastAPI-based server for non-Python stacks to access compression via API.

Related

  • Project
    microsoft/LLMLingua

    A prompt compression toolkit that removes non-essential tokens from LLM prompts to reduce costs, bypass token limits, and improve long-context performance.

  • Project
    yvgude/lean-ctx

    A context engineering layer for AI coding agents that compresses inputs and shell outputs to reduce token costs and extend context windows.

  • Project
    larsderidder/context-lens

    A local proxy that captures LLM API calls from AI coding tools to provide a detailed visual breakdown of context window composition and token costs.