Scaling Laws and Compute
Capabilities of modern AI improve predictably with model size, dataset size, and training compute - a finding with deep implications for research, economics, and policy.
Key facts
- Kaplan et al. (2020) established neural scaling laws across 7+ orders of magnitude.
- Chinchilla (2022) revised optimal model/data scaling to ~20 tokens per parameter.
- Frontier training compute has grown ~4–5x per year since 2010.
- Frontier 2025 training runs are estimated at ~10^26 FLOPs.
- Inference-time scaling emerged as a second productive axis in 2024.
- High-quality public text data is projected to be exhausted between 2026 and 2032.
Kaplan and Chinchilla Laws
Kaplan et al. (OpenAI, 2020) showed that test loss decreases as a smooth power law in compute, parameters, and data - across more than seven orders of magnitude.
Chinchilla (Hoffmann et al., DeepMind, 2022) refined this: for a fixed compute budget, optimal performance requires scaling model size and training tokens together, roughly 20 tokens per parameter. Most pre-Chinchilla models were significantly under-trained.
The Compute Trajectory
Compute used for frontier training has grown roughly 4–5x per year since 2010 - far faster than Moore's Law (~1.4x per year). Epoch AI tracks this trend across hundreds of notable training runs.
Frontier 2025 training runs are estimated at 10^26 FLOPs, costing hundreds of millions of dollars. Single-cluster scale has surpassed 100,000 H100-equivalent GPUs (xAI Colossus, OpenAI/Microsoft Stargate plans).
Inference-Time Scaling
Since 2024, a second scaling axis has emerged: spending more compute at inference time produces better answers on hard problems. OpenAI's o1/o3 and DeepSeek-R1 demonstrate that reasoning-trained models scale predictably with thinking tokens.
This shifts the economic frontier: inference compute now rivals training compute in importance, and per-query cost can vary by orders of magnitude depending on reasoning depth.
Limits to Scaling
Data exhaustion: high-quality public text is finite; Villalobos et al. (Epoch AI, 2024) project depletion of high-quality language data by 2026–2032. Synthetic data and multimodal sources extend the runway.
Energy and chips: a single 100K-GPU cluster draws ~150 MW. US grid expansion and TSMC advanced-node capacity are now binding constraints. Capital availability - frontier labs are raising tens of billions annually - completes the limiting trio.
Frequently asked
Will scaling alone produce AGI?
+
Contested. Some researchers project continued capability gains will produce AGI by scaling alone; others believe new architectural and learning-algorithm ideas will be required. Both camps include serious researchers.
How much compute does GPT-4 use?
+
Public estimates suggest ~2x10^25 FLOPs of training compute, equivalent to tens of thousands of A100/H100 GPUs over months and a total cost in the high tens of millions of dollars.
What is inference-time scaling?
+
Using more compute per query - by sampling many candidates, running explicit chain-of-thought, or tree search - to improve answer quality on hard problems. Now standard in reasoning models.
Sources & further reading
Learn & build with AI
Reference books and hardware picks for practitioners and curious readers. As an Amazon Associate BRAINMATTER earns from qualifying purchases at no extra cost to you.
Sponsor this work
BRAINMATTER is independent and reader-supported. Everything here stays free, ad-light, and open to everyone. A one-time sponsorship covers research time, editorial review, and hosting so we can keep publishing without paywalls.
Continue in this series
Foundations
Machine Learning: The Foundations
Neural Networks
Deep Learning: Hierarchical Representation from Raw Data
Architecture
The Transformer Architecture
LLMs
Large Language Models: How They Work and Where They Fail
Cross-Modal
Multimodal AI: Text, Vision, Audio, Video, and Action
Learning from Reward
Reinforcement Learning: From AlphaGo to RLHF