Abstract

As embedding models proliferate across retrieval-augmented generation pipelines and semantic search products, the memory footprint of approximate nearest neighbor (ANN) indexes has become a first-order infrastructure cost. Billion-scale corpora that would require hundreds of gigabytes under naive float32 HNSW indexes can be served within a fraction of that with modern quantization and graph compression techniques. This post examines scalar and product quantization, the RaBitQ and Matryoshka approaches, and what the current generation of vector databases - Weaviate, Qdrant, Milvus - exposes to practitioners navigating the recall-latency-memory tradeoff.

Quantization Fundamentals

Scalar quantization (SQ8, SQ4) maps each float32 dimension to an 8-bit or 4-bit integer, reducing a 1536-dimensional OpenAI text-embedding-3-large vector from 6 KB to 1.5 KB or 768 bytes. The compression is lossless in the indexing topology - the HNSW or IVF graph structure is unchanged - but introduces quantization error that slightly degrades recall. Product quantization (PQ) goes further: it partitions the vector into subspaces and codes each subspace independently using a learned codebook, achieving 16x to 64x compression at the cost of more pronounced recall degradation. The practical tradeoff is well-characterized by the recall@10 vs memory/query benchmarks that Qdrant and Milvus publish with each major release.

Graph-Aware Compression with RaBitQ

RaBitQ, published in mid-2024 and integrated into several production systems by late 2025, applies randomized binary quantization with a reranking step using the original float32 vectors for candidate refinement. The key insight is that the binary quantized graph can be searched cheaply to produce a shortlist, and the expensive distance computation only happens for the top-k candidates against the uncompressed originals. This two-phase approach achieves recall within 1-2% of full-precision HNSW while reducing peak memory by 4-8x. Qdrant’s scalar quantization with oversampling follows similar logic, though without the binary encoding.

Matryoshka Representation Learning

Matryoshka embeddings, popularized by the Matryoshka Representation Learning (MRL) paper and adopted by OpenAI in text-embedding-3, allow truncating the embedding vector to a shorter dimension without retraining the model. A 1536-dimensional vector can be truncated to 512 or 256 dimensions at query time, trading recall for speed and memory. This makes dimensional truncation a first-class operator in the retrieval stack: indexes can be built on the truncated form, and the full vector retained only for reranking. The practical impact on recall depends on the domain, but published figures for general English retrieval benchmarks show less than 5% NDCG degradation at 512 dimensions.

Operational Considerations

None of these techniques are free of operational complexity. SQ8 with oversampling requires tuning the oversampling factor per dataset - too low and recall suffers, too high and the reranking step erases the latency gain. PQ codebooks are trained on a sample of the data and must be retrained when the embedding distribution shifts significantly, which happens on model upgrades. Matryoshka truncation is only valid if the embedding model was trained with MRL; applying truncation to a standard model produces garbage. Before committing to any compression strategy, practitioners should benchmark on a representative sample of their own data rather than relying on published figures from different domains.