jia-gao/leanctx

Drop-in prompt compression for production LLM apps. Cut your token bill 40-60% without changing your code. Python SDK, LLMLingua-2, MIT.

What it solves

leanctx is a drop-in prompt compression library designed to reduce LLM input token costs by 40-60% without requiring changes to existing application code. It specifically targets dynamic per-query content—such as chat histories, freshly retrieved RAG documents, and tool outputs—which cannot be optimized by provider-level prompt caching.

How it works

The library acts as a wrapper around existing LLM SDKs (OpenAI, Anthropic, Gemini). It intercepts requests and passes them through a configurable compression pipeline:

  1. Classification: It identifies content types (e.g., code, error logs, prose).
  2. Routing: Based on configuration, it routes content to different compressors:
    • Verbatim: Preserves critical data (code, stack traces, tool calls) byte-identically.
    • Lingua: Uses a local LLMLingua-2 model to compress prose at zero marginal cost.
    • SelfLLM: Uses a cheap LLM (like GPT-4o-mini) to summarize content.
  3. Execution: The compressed prompt is then sent to the actual provider API.

Who it’s for

Developers building production LLM applications, particularly those dealing with high token bills from:

  • RAG applications with large retrieved documents.
  • Long-running conversational agents (e.g., using LangChain or CrewAI).
  • Document-processing pipelines.
  • Coding agents with extensive tool-call histories.

Highlights

  • Drop-in Integration: Replaces standard SDK imports with leanctx wrappers for minimal code changes.
  • Local Compression: Supports LLMLingua-2 for local, private compression that doesn't send data to third-party compression APIs.
  • Content-Aware: Programmatically ensures that code and tool-call payloads remain untouched while compressing logs and prose.
  • Observability: Integrated OpenTelemetry support for tracking token savings, costs, and latency.
  • Flexible Deployment: Available as a Python library or an HTTP sidecar for non-Python stacks.