tonl-dev/tonl

What it solves

TONL (Token-Optimized Notation Language) is a data serialization format and platform designed to reduce the size of structured data, specifically to lower token costs when feeding data into Large Language Models (LLMs). It provides a human-readable alternative to JSON that is significantly more compact in both bytes and tokens while maintaining full bidirectional compatibility with JSON.

How it works

TONL uses a compact notation that replaces repetitive JSON keys and structural overhead with a schema-like header (e.g., users[3]{id:u32,name:str,role:str}:) followed by raw data values. It includes a comprehensive TypeScript API for encoding, decoding, and querying data using JSONPath-like syntax. To further reduce size, it employs advanced optimization strategies such as dictionary encoding, delta encoding, and bit packing.

Who it’s for

• LLM Developers: Those looking to reduce API costs and increase context window efficiency by compressing structured data in prompts.
• Data Engineers: Users needing a fast, lightweight, and human-readable serialization format for configuration files or API responses.
• Frontend Developers: Developers needing a small, zero-dependency library for data manipulation in the browser.

Highlights

• Token Efficiency: Reduces JSON size and LLM token usage by 32-45%.
• Powerful Querying: Supports JSONPath queries, filter expressions, and tree traversal.
• Advanced Compression: Includes adaptive optimization, numeric quantization, and run-length encoding.
• Production Ready: Zero runtime dependencies, 100% TypeScript, and extensive test coverage (2,300+ tests).
• Developer Tooling: Features an interactive CLI dashboard with themes, a schema validation language (TSL), and a VS Code extension.