misa77 0.2.0: High-Throughput LZ-Based Decompression Codec

misa77 0.2.0: High-Throughput LZ-Based Decompression Codec

misa77 0.2.0 is an LZ-based compression codec optimized for "write-once, read-many" scenarios. It prioritizes extreme single-threaded decompression throughput and constant memory usage over compression speed, making it suitable for applications where data is compressed once and read frequently, such as game engine assets or boot firmware.

Performance Benchmarks

On x86-64 (Intel Core i7-14650HX), misa77 0.2.0 demonstrates significant decompression speed advantages over LZ4 and other fast-decode codecs.

Silesia Corpus Results

In tests using the Silesia Corpus, misa77 0.2.0 (Level 0) achieved a decompression speed of 5219 MB/s, compared to 2505 MB/s for LZ4 1.10.0. This represents more than a 2x speedup while maintaining a competitive compression ratio (42.64 for misa77 vs 47.59 for LZ4).

enwik8 Results

For the enwik8 dataset, misa77 0.2.0 (Level 0) reached 4802 MB/s decompression throughput, while LZ4 1.10.0 reached 2355 MB/s.

Key Performance Characteristics

  • Pareto Frontier: misa77 typically sits on the Pareto frontier for decompression throughput versus compression ratio across most data shapes.
  • Data-Dependent Speed: Decompression throughput is higher on highly compressible files. Spending more effort during compression (higher levels) often leads to better decompression throughput.
  • Trade-offs: The primary trade-off for this decompression speed is slow compression throughput. For example, on the Silesia corpus, misa77 (Level 0) compresses at 54.5 MB/s, whereas LZ4 compresses at 371 MB/s.

Technical Specifications and Constraints

Memory and Architecture

  • Memory Usage: The codec maintains constant memory use regardless of input size. Decompression requires 0 MB of additional memory, and compression requires ≤ 5 MB across all modes.
  • Architecture Support: It is designed for little-endian 64-bit systems. On x86-64, it utilizes AVX2/SSE2 at runtime. Other architectures use a portable path that is auto-vectorizable by compilers (e.g., Apple ARM).

Compression Levels

As of v0.2.0, two primary effort levels are available:

  • Level 0: Optimized for maximum decode throughput with a slightly worse compression ratio.
  • Level 1 (Default): Optimized for a better compression ratio with slightly lower decode throughput.

CLI and Integration

The misa CLI provides three primary subcommands for file-based operations:

  • compress: Compresses a file to .misa77 format. Supports -l N for level selection and experimental flags like --adaptive (autotuning based on input) and --yolo (high-effort, decode-optimized).
  • decompress: Restores a .misa77 file to its original state.
  • suggest: Analyzes a file to produce a .misap file containing tuned parameters for optimal compression.

Current Status and Limitations

Because misa77 is in version 0.x.y, it is currently considered experimental. The author explicitly notes the following limitations:

  • Format Stability: The stream format may change unexpectedly.
  • Security: The decoder assumes input is a valid misa77 stream. Invalid or malicious input results in undefined behavior (UB), as the codec is not yet hardened against corrupted data.
  • Dependencies: The CLI requires a POSIX environment (Linux, macOS) and a C++20 compiler.

Community Insights

Discussion among technical peers highlights both the potential and the risks of the current implementation. Some users noted that the lack of robustness against corrupted data puts it in a different class of algorithm compared to hardened production codecs.

"If you are not robust to corrupted/malicious data, it is really in a different class of algorithm and it is hard to compare speeds directly."

Other contributors pointed out that the high decompression speed on highly compressible data is a known trade-off in performance engineering, where the encoder spends more time organizing the format to be friendlier to memcpy operations, thereby reducing the CPU overhead during decompression.

Sources