Embedding Quantization: Binary and Scalar Techniques for Faster, Cheaper Retrieval

TL;DR: Hugging Face introduced binary and scalar (int8) embedding quantization, which compresses embeddings by 32× or 4× respectively, cuts memory and storage costs dramatically, and speeds up retrieval up to 45× while preserving 96%–99% of original performance.

Why Embeddings Matter and How They Scale

Embeddings turn text, images, audio, and other data into high‑dimensional vectors that enable similarity search, recommendation, clustering, and many downstream NLP tasks. State‑of‑the‑art models often emit 1024‑dimensional float32 vectors, requiring 4 bytes per dimension. Storing 250 M such vectors consumes ~1 TB of RAM, leading to multi‑thousand‑dollar monthly cloud bills. The blog quantifies these costs for several popular models, showing that a 1024‑dimensional model can cost over $3,600 / mo on AWS x2gd instances.

Quantization vs. Dimensionality Reduction

Traditional scaling approaches use dimensionality reduction (e.g., PCA) or Matryoshka Representation Learning (MRL), which truncate dimensions but can hurt performance. Embedding quantization instead reduces the precision of each dimension after the model has produced the embedding, offering a complementary path to cheaper retrieval.

Binary Quantization

Binary quantization converts each float32 value to a single bit by thresholding at zero. This yields a 32× reduction in storage (e.g., a 1024‑dimensional vector becomes 1024 bits, packed into 128 bytes). Retrieval uses Hamming distance, which can be computed in just two CPU cycles, delivering massive speedups.

Implementation in Sentence‑Transformers

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("mixedbread-ai/mxbai-embed-large-v1")
# Direct binary encoding
binary_embeddings = model.encode(
    ["I am driving to the lake.", "It is a beautiful day."],
    precision="binary",
)

The resulting binary_embeddings have shape (2, 128), int8 dtype, and occupy 256 bytes versus 8 192 bytes for the original float32 embeddings.

Support in Vector Databases

Binary indexes are available in Faiss, USearch, Vespa AI, Milvus, Qdrant, and Weaviate, enabling drop‑in replacement of existing pipelines.

Scalar (int8) Quantization

Scalar quantization maps the continuous float32 range of each dimension to 256 discrete int8 levels (‑128 to 127). This yields a 4× storage reduction while retaining finer granularity than binary. Calibration on a large embedding set is required to compute per‑dimension min/max ranges.

Implementation in Sentence‑Transformers

from sentence_transformers import SentenceTransformer, quantize_embeddings
from datasets import load_dataset
model = SentenceTransformer("mixedbread-ai/mxbai-embed-large-v1")
corpus = load_dataset("nq_open", split="train[:1000]")["question"]
calibration_embeddings = model.encode(corpus)
embeddings = model.encode(["I am driving to the lake.", "It is a beautiful day."])
int8_embeddings = quantize_embeddings(
    embeddings,
    precision="int8",
    calibration_embeddings=calibration_embeddings,
)

The int8_embeddings retain the original 1024‑dimensional shape but use only 2 048 bytes.

Support in Vector Databases

Scalar quantization is supported (directly or indirectly) in Faiss (IndexHNSWSQ), USearch, Vespa AI, OpenSearch, ElasticSearch, Milvus (IVF_SQ8), and Qdrant.

Combining Binary and Scalar Quantization

A two‑stage pipeline can achieve the best of both worlds:

1. Encode the query with a high‑quality model (e.g., mxbai-embed-large-v1).
2. Quantize the query to binary and search a binary index (≈5 GB for 41 M Wikipedia passages).
3. Load the top‑k candidates from an int8 index stored on disk (≈48 GB).
4. Rescore those candidates with the original float32 query against the int8 embeddings.
5. Return the final top‑k results. This approach reduces memory to ~5 GB and disk to ~52 GB, compared with ~200 GB required for full‑precision retrieval.

Experimental Results

Retrieval Performance

Model	Dim	Storage (250 M)	MTEB Retrieval NDCG@10	% of Float32
mxbai-embed-large-v1 (float32)	1024	953.67 GB $3 623/mo	54.39	100 %
mxbai-embed-large-v1 (int8)	1024	238.41 GB $905/mo	52.79	97 %
mxbai-embed-large-v1 (binary)	1024	29.80 GB $113/mo	52.46	96.45 %
all-MiniLM-L6-v2 (binary)	384	11.18 GB $42/mo	39.07	93.79 %

Key observations:

• Int8 quantization often retains >94 % of performance while cutting storage by 4×.
• Binary quantization retains ~96 % for large‑dimensional models and can even outperform int8 for some small models (e.g., all-MiniLM-L6-v2).
• Performance varies by model; calibration data quality and dimension collapse can affect results.

Influence of Rescoring

• Binary rescoring (re‑ranking the top‑k binary results with the original float query) lifts performance from 92.5 % to 96.5 % of the baseline.
• For int8, increasing the rescore_multiplier (retrieving more candidates before rescoring) improves retention, reaching ~99 % at a multiplier of 4–5.

Retrieval Speed

On a GCP a2-highgpu-4g CPU‑only exact search:

Quantization	Min Speedup	Mean Speedup	Max Speedup
float32	1×	1×	1×
int8	2.99×	3.66×	4.8×
binary	15.05×	24.76×	45.8×
Binary quantization therefore offers order‑of‑magnitude latency reductions.

Summary of Trade‑offs

Metric	float32	int8/uint8	binary/ubinary
Memory & Index Size	1×	4× smaller	32× smaller
Retrieval Speed	1×	up to 4× faster	up to 45× faster
Performance Retention	100 %	~99 %	~96 %

Demo and Practical Scripts

A live demo (link) shows retrieval over 41 M Wikipedia passages using 5 GB RAM and 52 GB disk, achieving the speedups above. The blog also provides three categories of ready‑to‑run scripts:

• Recommended Retrieval – combines binary search with int8 rescoring.
• Usage – shows how to call semantic_search_faiss or semantic_search_usearch with quantized embeddings.
• Benchmarks – measures speed and accuracy for each quantization mode.

Future Directions

• Explore sub‑int8 quantization (e.g., 4‑bit or 2‑bit buckets) for even tighter compression.
• Combine quantization with Matryoshka Representation Learning to first truncate dimensions then quantize, potentially achieving 32×–256× speedups with modest quality loss.
• Integrate a third‑stage cross‑encoder reranker after binary + int8 stages for state‑of‑the‑art retrieval at low latency and cost.

---

Citation

@article{shakir2024quantization,
  author = {Aamir Shakir and Tom Aarsen and Sean Lee},
  title = {Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval},
  journal = {Hugging Face Blog},
  year = {2024},
  note = {https://huggingface.co/blog/embedding-quantization}
}