jia-gao/leanctx

Drop-in prompt compression for production LLM apps. Cut your token bill 40-60% without changing your code. Python SDK, LLMLingua-2, MIT.

What it solves

leanctx 是一个即插即用的提示词压缩库，旨在无需更改现有应用程序代码的情况下，将 LLM 输入 Token 成本降低 40-60%。它专门针对动态的单次查询内容——例如聊天记录、新检索的 RAG 文档和工具输出——这些内容无法通过提供商层级的提示词缓存来优化。

How it works

该库作为现有 LLM SDKs (OpenAI, Anthropic, Gemini) 的包装器。它拦截请求并将其通过一个可配置的压缩管道：

  1. Classification: It identifies content types (e.g., code, error logs, prose).
  2. Routing: Based on configuration, it routes content to different compressors:
    • Verbatim: Preserves critical data (code, stack traces, tool calls) byte-identically.
    • Lingua: Uses a local LLMLingua-2 model to compress prose at zero marginal cost.
    • SelfLLM: Uses a cheap LLM (like GPT-4o-mini) to summarize content.
  3. Execution: The compressed prompt is then sent to the actual provider API.

Who it’s for

开发者正在构建生产环境的 LLM 应用程序，特别是那些处理以下情况产生的的高 Token 账单的开发者：

  • 具有大型检索文档的 RAG 应用。
  • 长时间运行的对话代理 (例如，使用 LangChain 或 CrewAI)。
  • 文档处理流水线。
  • 具有广泛工具调用历史的编码代理。

Highlights

  • Drop-in Integration: Replaces standard SDK imports with leanctx wrappers for minimal code changes.
  • Local Compression: Supports LLMLingua-2 for local, private compression that doesn't send data to third-party compression APIs.
  • Content-Aware: Programmatically ensures that code and tool-call payloads remain untouched while compressing logs and prose.
  • Observability: Integrated OpenTelemetry support for tracking token savings, costs, and latency.
  • Flexible Deployment: Available as a Python library or an HTTP sidecar for non-Python stacks.