jia-gao/leanctx

Drop-in prompt compression for production LLM apps. Cut your token bill 40-60% without changing your code. Python SDK, LLMLingua-2, MIT.

What it solves

leanctx 是一个即插即用的提示詞壓縮庫，旨在於無需更改現有應用程式碼的情況下，將 LLM 輸入 Token 成本降低 40-60%。它特別針對動態的單次查詢內容——例如聊天記錄、新檢索的 RAG 文件和工具輸出——這些內容無法透過提供商層級的提示詞快取來優化。

How it works

該庫作為現有 LLM SDKs (OpenAI, Anthropic, Gemini) 的包裝器。它攔截請求並將其通過一個可配置的壓縮管道：

  1. Classification: It identifies content types (e.g., code, error logs, prose).
  2. Routing: Based on configuration, it routes content to different compressors:
    • Verbatim: Preserves critical data (code, stack traces, tool calls) byte-identically.
    • Lingua: Uses a local LLMLingua-2 model to compress prose at zero marginal cost.
    • SelfLLM: Uses a cheap LLM (like GPT-4o-mini) to summarize content.
  3. Execution: The compressed prompt is then sent to the actual provider API.

Who it’s for

開發人員正在構建生產環境的 LLM 應用程式，特別是那些處理以下情況產生的高 Token 成本的開發人員：

  • 具有大型檢索文檔的 RAG 應用程式。
  • 長時間運行的對話代理 (例如，使用 LangChain 或 CrewAI)。
  • 文件處理管道。
  • 具有廣泛工具調用歷史的編碼代理。

Highlights

  • Drop-in Integration: Replaces standard SDK imports with leanctx wrappers for minimal code changes.

  • Local Compression: Supports LLMLingua-2 for local, private compression that doesn'0t send data to third-party compression APIs.

  • Content-Aware: Programmatically ensures that code and tool-call payloads remain untouched while compressing logs and prose.

  • Observability: Integrated OpenTelemetry support for tracking token savings, costs, and latency.

  • Flexible Deployment: Available as a Python library or an HTTP sidecar for non-Python stacks.

         

    ext{...}

解決問題

How it works

leanctx 是一个即插即用的提示詞壓縮庫，旨在於無需更改現有應用程式碼的應用程式，旨在於無需更改現有應用程式碼的應用程式，旨在於無需更改現thoughtful_thought: 32 tokens.```jsonstring{