Abstract
Neural scaling laws describe how test loss changes as a power function of model size, dataset size, and compute budget, allowing practitioners to predict performance at scales not yet trained. The Chinchilla paper (Hoffmann et al., DeepMind, 2022) revised earlier compute-optimal recipes by showing that prior large language models were significantly under-trained relative to their parameter counts. In the three years since, the implications of scaling laws have spread well beyond language models - to vision, multimodal, and even reinforcement learning settings - while new research has interrogated whether the power-law fits hold at extreme scale and what they imply for data efficiency when tokens are finite.
The Chinchilla Shift and Its Aftermath
The original Kaplan et al. scaling laws (OpenAI, 2020) suggested that for a fixed compute budget, model size should grow faster than dataset size. Chinchilla reversed this prescription: doubling model parameters should roughly double training tokens to stay compute-optimal. Empirically, a 70B-parameter model trained on 1.4 trillion tokens matched or exceeded the 280B Gopher model on most benchmarks at lower inference cost. This finding shifted the field toward leaner, more thoroughly trained models - Llama 2 and Mistral 7B being prominent examples of the philosophy applied in open releases. The practical consequence is that data curation and collection now compete with architecture scaling as a bottleneck, since the Chinchilla recipe demands more tokens than many domains can naturally supply.
Data Efficiency and the Finite-Data Regime
When the available corpus is bounded, compute-optimal scaling laws no longer dictate the training recipe. Repeated-epoch training violates the single-pass assumption underlying most scaling law derivations, and empirical results on code and scientific literature show that loss curves flatten or slightly worsen after 2-4 epochs at standard learning rates, depending on dataset diversity. Muennighoff et al. (2023) showed that for sufficiently data-limited regimes, repeating data fewer than four times costs less than the corresponding amount of unique data would at compute-optimal token counts. This nuance matters for specialized domains - genomics, legal text, low-resource languages - where unique tokens are scarce. Data mixing strategies, curriculum ordering, and deduplication pipelines (using MinHash or SimHash at scale) all interact with the effective data budget in ways that straightforward power-law fits do not capture.
Scaling Laws Beyond Language
Vision models follow analogous but quantitatively distinct scaling laws. Zhai et al. (Google Brain, 2022) fitted power laws to ViT training across five orders of magnitude in compute, finding exponents that differ meaningfully from the NLP case, partly because image datasets have different effective diversity per sample than text corpora. For multimodal contrastive models, scaling behavior depends jointly on the image and text encoders, and the optimal allocation between them is not well characterized. In reinforcement learning, scaling laws for world models and policy networks are an active area with few consensus results, partly because reward signal density adds a variable orthogonal to data volume.
Practical Implications for Model Development
Scaling law estimates are most useful as guardrails against obvious misallocation - training a 7B model on 10B tokens when 100B would be compute-optimal - rather than precise performance predictors. Loss measured on held-out tokens does not always translate monotonically to downstream task performance, particularly for few-shot and chain-of-thought tasks that emerge discontinuously with scale. Practitioners at organizations without frontier compute budgets can use scaling law fits on small proxy experiments to rank architectural variants before committing to full training runs, a technique referred to as hyperparameter transfer or mu-parameterization (Yang et al., Microsoft).