Breaking the 1.58-bit Barrier for Ternary LLMs

Researchers have developed a new weight storage layout called BITCOS that allows ternary Large Language Models (LLMs) to be stored more compactly than the theoretical information-theoretic limit of 1.585 bits per weight. By exploiting the fact that zeros are disproportionately common in ternary weights, BITCOS reduces the effective bit-width to as low as 1.485 bits per weight in the sparsest models.

The 1.58-bit Barrier and Current Limitations

Ternary LLMs store weights as one of three symbols: $\text{−1}, 0, +1$. The theoretical minimum storage cost is $\log_2 3 \approx 1.585$ bits per weight. In current production deployments, the prevailing format is "five-trit packing," which packs five ternary weights into one byte. Due to power-of-two group sizes, this results in an effective storage bit-width of 1.625 bits per weight.

This existing approach treats the three symbols as equiprobable, assuming a distribution of roughly 33.3% for each symbol. However, the researchers measured the actual symbol distribution of 29 different ternary LLM models and found that zeros account for up to 51.5% of all weights.

BITCOS: Distribution-Adaptive Layout

To leverage this high zero-density, the researchers introduced BITCOS, a distribution-adaptive layout. Instead of a fixed-bit packing scheme, BITCOS uses a two-part structure:

Dense Presence Bitmap: A bitmap that indicates whether a weight is zero or non-zero.
Compacted Sign Vector: A compacted vector that stores only the signs of the non-zero weights.

The storage cost of BITCOS is defined as $2 - z$ bits per weight element, where $z$ is the zero density in the model's weights. As zero density increases, the storage cost decreases.

Performance and Hardware Optimization

// a single section on performance metrics

BITCOS outperforms five-trit packing in 26 of the 29 tested models. In the sparsest models, it reaches a storage efficiency of 1.485 bits per weight. The authors provide optimized unpacking sequences for modern processors and GPUs, including AVX-512, AVX2, and Intel Xe2 GPUs.

Measured against state-of-the-art ternary matrix-vector multiplication kernels, BITCOS provides the following gains:

Realized Gain: Up to 1.28$ imes$ improvement in matrix-vector multiplication kernels.
CPU Decode Throughput: Up to 1.18$ imes$ improvement on client and server CPUs.
GPU Decode Throughput: Up to 1.27$ imes$ improvement on Intel Xe2 GPUs.

Community Insights and Counterpoints

Discussion among technical users on Hacker News suggests that this research is particularly relevant for edge computing and embedded systems where VRAM and RAM limits are strict.

One user noted that while the efficiency gains are significant, the alternative of vector quantization or trellis-based methods for Post-Training Quantization (PTQ) might be more effective in this region. Another user suggested that arithmetic coding could potentially squeeze out even more bits, though perhaps at the cost of higher decompression overhead.

"If ternary llms work out and are baked into hardware as custom silicon I bet they'll be shockingly efficient."

"This could drastically shrink LLMs for embedded systems, making them truly portable."

Implementation Details

The researchers' findings indicate that that the 1.58-bit limit is only a limit for equiprobable symbols. By treating the weight distribution as a distribution-adaptive layout, BITCOS BITCOS allows for a model to be stored in a memory format that is usable directly as an in-memory format, rather than just a storage or transfer format.