ctrlb-decompose: Log Compression and Pattern Extraction for LLMs
ctrlb-decompose: Log Compression and Pattern Extraction for LLMs
Log Compression for LLM Analysis
ctrlb-decompose is a high-performance tool designed to strip noise from raw logs, transforming millions of lines into a handful of actionable structural patterns. This process reduces token consumption and noise when feeding logs into Large Language Models (LLMs), allowing models to reason over the structural essence of the logs rather than individual, repetitive lines.
Two-Stage Normalization and Clustering Pipeline
ctrlb-decompose processes logs in a single streaming pass with a minimal memory footprint using a two-stage pipeline:
Stage 1: CLP Encoding
The tool uses Compressed Log Processor (CLP) encoding to normalize variable tokens into typed placeholders. This ensures that structurally identical lines produce the same "logtype" regardless of the specific values present. For example, a line like Request from 10.0.1.15 completed in 45ms status=200 is normalized to Request from <dict> completed in <float>ms status=<int>.
Stage 2: Drain3 Clustering
Following encoding, the Drain3 algorithm builds a prefix tree over logtypes to group them by token similarity (default threshold of 0.4). Where tokens diverge, the tool inserts <*> wildcards to create a template. This process is incremental, meaning each line is processed once without requiring a second pass.
Variable Classification and Semantic Typing
Extracted variables are classified into semantic types to provide richer analysis and statistics. The tool detects the following types:
| Type | Example | Detection Method |
|---|---|---|
IPv4 / IPv6 |
10.0.1.15 |
CIDR pattern match |
UUID |
550e8400-e29b-... |
8-4-4-4-12 hex format |
Duration |
45ms, 3.2s |
Numeric + time unit suffix |
HexID |
0x1a2b3c |
4+ hex digits |
Integer |
200 |
Parses as i64 |
Float |
3.14 |
Contains ., parses as f64 |
Enum |
ERROR |
Low cardinality (<=20 unique, top-3 >= 80%) |
Timestamp |
2024-01-15T14:22:01Z |
RFC 3339 pattern |
String |
any other value | Fallback |
Memory Efficiency and Statistical Accumulation
To maintain a low memory footprint, ctrlb-decompose utilizes several probabilistic data structures:
- Drain3 Clusters: Managed with O(k) complexity and LRU eviction (10k max).
- Quantiles: Uses DDSketch to maintain fixed ~200 bytes per numeric slot without storing raw values.
- Cardinality: Employs HyperLogLog++ for high-cardinality variables (~200 bytes per variable).
- Examples: Uses reservoir sampling to maintain a bounded buffer of example lines per pattern.
Output Formats and Integration
ctrlb-decompose supports three primary output formats to suit different workflows:
- Human (
--human): The default output, featuring ANSI colors and visual bars for terminal investigation. - LLM (
--llm): A compact, token-efficient Markdown format specifically designed for LLM consumption. - JSON (
--json): A structured format for programmatic consumption.
Claude Code Plugin
The tool integrates with Claude Code via a plugin. Users can install it using /plugin marketplace add ctrlb-hq/ctrlb-decompose and then analyze logs using natural language commands such as "Analyze the errors in /var/log/app.log."
Community Insights
Users on Hacker News highlighted the tool's adherence to the Unix philosophy by focusing on a single decomposition step without requiring a full ecosystem adoption. Some suggested extending the tool's capabilities to integrate with OpenTelemetry for converting patterns into metrics, logs, or traces to reduce data shipping costs.
"Finally, a tool that tackles this problem without forcing you to adopt an entire ecosystem. I appreciate how this leans into the Unix philosophy by just focusing on the decomposition step and doing it well."
Installation
ctrlb-decompose is available via Homebrew for macOS (brew install ctrlb-decompose), as a .deb package for Debian/Ubuntu, or can be built from source using Cargo.