Abstract
Post-training quantization of large language models to 4-bit precision has become standard practice for edge and resource-constrained deployment, with tools like GPTQ, AWQ, and llama.cpp making 4-bit inference broadly accessible. The next frontier is sub-4-bit quantization, specifically 3-bit and 2-bit regimes, where the accuracy-compression tradeoff becomes dramatically harder. Post-training methods degrade sharply below 4 bits; quantization-aware training (QAT), which simulates quantization noise during the forward pass and uses straight-through estimators for gradient flow, is increasingly viewed as the path to viable sub-4-bit models. This article examines the state of QAT for LLMs in mid-2025, including practical recipes, remaining failure modes, and the emerging consensus on where QAT is and is not worth the training cost.
Why Post-Training Quantization Breaks Below 4 Bits
Post-training quantization works by finding good weight groupings and scale factors after training is complete, without updating the weights themselves. At 4 bits, the quantization error per weight is small enough that calibration-based rounding correction (as in GPTQ) can absorb most of the accuracy loss. At 3 bits and especially 2 bits, the quantization grid is coarse enough that no post-hoc rounding strategy can recover the information destroyed by discretization. The problem is compounded by LLM weight distributions that contain outliers: a small fraction of weights take values much larger than the bulk of the distribution, and these outliers force the quantization grid to allocate range at the expense of precision for typical weights.
QAT Recipes for LLMs
QAT for LLMs differs from QAT for vision models in several ways. LLMs are expensive to train from scratch with quantization simulation; in practice, QAT is applied as a fine-tuning stage on a pretrained float model. The straight-through estimator allows gradients to flow through the rounding operation, allowing the model to adapt its weight distribution to quantization-friendly shapes. QuIP# (from Cornell) and AQLM (from IST Austria) are two of the more carefully studied approaches as of mid-2025, both targeting 2-bit compression through a combination of incoherence preprocessing, learned codebooks, and calibration fine-tuning. Both report perplexity within 1-2 points of the float baseline on standard benchmarks for 7B-class models, a result that was not achievable with post-training methods alone.
A critical practical detail is the calibration dataset. QAT for LLMs uses a small subset of training data (typically 512 to 4096 sequences) to drive the fine-tuning phase. The distribution of this calibration set matters: domain mismatch between calibration and deployment leads to quantization error patterns that generalize poorly.
Failure Modes
Sub-4-bit QAT models degrade non-uniformly across task types. Arithmetic and formal reasoning tasks are more sensitive to precision loss than open-ended generation; a 2-bit model that achieves near-float perplexity on text continuation may show larger regression on GSM8K-style benchmarks. Attention weight quantization is more fragile than weight quantization for MLP layers; most practical QAT recipes quantize weights but keep activations in higher precision, which limits hardware efficiency gains. True weight-and-activation quantization to 2 bits remains largely unsolved for 7B+ models.
When QAT Is Worth the Cost
For teams deploying models on memory-constrained edge hardware (mobile NPUs, single-GPU consumer setups), the additional compute cost of a QAT fine-tuning run can be justified by meaningful throughput and footprint improvements. For server-side inference where 4-bit is already achievable, the marginal gain from 2-bit is smaller and the training cost is harder to justify unless the deployment scale is large. The practical case for QAT below 4 bits is strongest for models under 13B parameters, where the fine-tuning cost is manageable and the memory savings are relatively large compared to available hardware budgets.