RTK Token Savings: Benchmarks Reveal Limited Cost Reduction in AI Coding

RTK fails to provide general cost savings for AI coding agents

Rust Token Killer (RTK) is designed to reduce the cost of AI coding by filtering and compressing terminal output before it reaches the AI agent. However, empirical benchmarks on Terminal-Bench 2.1 reveal that RTK does not consistently lower total costs. For some models, it can actually increase expenses by forcing the agent to take more turns to complete a task, effectively canceling out any savings gained from shorter individual prompts.

Benchmark Methodology: Terminal-Bench 2.1

To evaluate the impact of RTK, researchers tested it using Claude Code (with Fable 5.0) and OpenCode (with DeepSeek V4 Pro 0813) across 1,740 attempts on Terminal-Bench 2.1. The benchmark focused on tasks that agents typically pass, as cost optimization is only relevant for successful completions.

Cost and Pass Rate Results

The results show a negligible or negative impact on total spending:

Claude Code (Fable 5.0): Total costs fell by 5% ($731 baseline vs $698 with RTK), but the majority of these savings were attributed to a single task (winning-avg-corewars). When excluding that outlier, savings were less than 1%.
OpenCode (DeepSeek V4 Pro 0813): Total costs rose by 5% ($51 baseline vs $54 with RTK). On a task-level measure, the average cost per task increased by 17%.

Pass rates remained largely stable, with a slight decrease (1% for Fable and 2% for DeepSeek) when using RTK.

Why 'rtk gain' is a misleading metric

RTK reports a metric called rtk gain, which calculates the difference between raw and filtered command output in bytes divided by four. This is not a count of billed tokens and does not reflect actual money saved.

Benchmarks revealed that rtk gain can be wildly inflated. For example, in one task, RTK credited 241 million tokens saved because it compared limited reads (using head) against the entire file size, even though the original command would never have returned the full file. Because rtk gain does not account for the agent's subsequent reactions or the cost of additional turns, it can make an expensive attempt appear optimized.

The 'Tokenflation' Effect: More Turns, Higher Costs

Reducing the size of a single turn's input does not guarantee a lower total bill. In agentic coding, context is often cached, making subsequent reads of terminal output significantly cheaper (1/10th the cost for Fable, 1/30th for DeepSeek).

When RTK compresses output, it can confuse the model or remove critical information, leading to the "tokenflation" problem: the agent takes more turns to reach the same conclusion. For DeepSeek, RTK attempts took more turns on 58 tasks, and 44 of those cost more overall. The average DeepSeek turn had 7% less input, but there were 18% more turns overall, resulting in a net increase in cost.

Technical Risks and Limitations

Tool Incompatibility and Loops

RTK rewrites shell commands, which can introduce bugs. One benchmark attempt entered a loop of 339 consecutive errors because rtk find did not support a specific flag that the standard find command did. The agent repeatedly tried the same failing command, resulting in a cost 9x higher than the baseline.

Bypass Mechanisms

RTK only acts on shell commands. Many AI coding platforms use separate tools for Read, Grep, and Glob operations, which bypass RTK entirely. Furthermore, frontier models are increasingly efficient at using the terminal themselves, often employing head, tail, or wc to limit output manually.

Community Insights and Alternatives

Industry practitioners and developers have raised several counterpoints and alternatives to raw output compression:

"If the output is not what it expects, an LLM may issue more tool calls than before, because it will assume the tool is broken... more tokens."

"I just spawn a subagent in the cheapest range (for example flash-lite) to summarise a tool use. It's the only way that has worked based on my benchmarks."

Some users suggest that RTK may still be useful for specific, highly verbose tools (like Maven or Cargo) where the output is consistently redundant, but agree that wrapping all terminal output is generally counterproductive. Others recommend using explicit agent rules (e.g., COMMAND 2>&1 | head -c 4000) to cap output without altering the command's behavior through a third-party wrapper.