How Compaction Works in Pi

Context Management in Coding Agents

Large Language Models (LLMs) operate within a fixed context window, which limits the amount of input they can process during a single request. In a coding agent session, the input grows continuously as the system prompt, tool definitions, conversation history, and tool outputs accumulate. Once this total exceeds the context window, the LLM will reject the request with a size error.

To prevent session failure, agents must either start a new conversation—which discards all accumulated context and prior decisions—or implement compaction, which creates a smaller, compressed representation of the existing history to make room for new messages.

Pi's Compaction Implementation

Pi implements compaction by summarizing older conversation content while preserving a specific budget of recent messages. This process is triggered automatically when the context limit nears the total size of the window, or manually via the /compact command.

The Compaction Process

Pi's compaction follows a specific workflow to ensure the agent retains critical information without wasting tokens:

Retention Budget: Pi retains a configurable number of recent messages (defaulting to 20,000 tokens, roughly 5 to 20 turns) unchanged.
Serialization: All messages preceding the retention cut-off point are extracted and serialized for summarization.
Specialized Request: Pi sends a standalone request to a "context summarization assistant" rather than the standard coding assistant. This allows Pi to use a different, potentially more cost-effective LLM model for the summary.
Structured Output: The compaction prompt explicitly requests a structured summary divided into three sections: goal, progress, and key decisions. This acts as a "handoff briefing" for the next phase of the conversation.

Integration and Portability

The resulting summary is stored as plain text within the session. This approach ensures that the compacted context remains readable to the user and portable across different LLM models, allowing users to switch models without losing the summarized history.

Impact on Prompt Caching

Prompt caching reduces costs and latency by allowing LLM providers to reuse the prefix of a conversation. However, caching requires an exact token-for-token prefix match.

Because compaction replaces a large block of older history with a new summary, it changes the prefix of the conversation. This effectively breaks the existing prompt cache, meaning all tokens following the summary—including the retained recent turns—must be recomputed during the first request after compaction. Once the new state is established, subsequent requests will benefit from caching again.

Alternative Context Strategies and Community Perspectives

While Pi uses LLM-based summarization, developers and users have proposed several alternative strategies for managing context overflow:

Pruning vs. Summarization

Some users argue that pruning—the deterministic removal of low-value messages—is superior to summarization.

"I find summarized conversations lead to more frustrating future chats because the LLM misses intent and or context."

Proposed pruning strategies include removing "thinking" traces, tool call outputs, and codebase exploration logs while keeping user messages and final assistant conclusions.

Architectural Alternatives

Nested Threading: One approach involves moving old messages into a sub-thread that summarizes itself, providing a clean parent thread while keeping the full history accessible in the child thread.
Dynamic Pruning: Some implementations use labels to collapse and expand summaries of tool calls and chat history dynamically.
KV Cache Manipulation: For those running local stacks, some suggest operating directly on the GPU to purge or replace old tool calls with summaries during inference, avoiding the need for a full new LLM request.
Handoffs: Rather than compacting, some prefer a "handoff" where the current LLM is instructed to summarize everything necessary for a brand new conversation session.

Limitations of Current Approaches

Critics of standard compaction note that it can be computationally expensive for local LLMs to parse 128k tokens just to generate a small summary. Additionally, some users report that if a tool-calling loop is long, the agent may not check the compaction limit until the loop finishes, potentially leading to Out-of-Memory (OOM) errors.