How Scaling Laws Will Determine AI's Future

The host describes the shift from GPT-2, with 1.2 billion parameters, to GPT-3, which was more than 100 times larger. A 2020 OpenAI paper by Jared Kaplan, Sam McCandlish, and colleagues found that increasing model size, training data, and compute brought steady performance gains. The host says those gains followed a power law and depended more on scale than on algorithm changes.

Google DeepMind's 2022 research examined how to allocate a compute budget between model size and training data. Its researchers trained more than 400 models and concluded that models such as GPT-3 were undertrained. Chinchilla, less than half GPT-3's size but trained on four times as much data, outperformed models more than twice its size at one point.

The host describes debate over whether larger, more expensive models are still delivering comparable capability gains. One unnamed participant says companies are increasing GPU use without seeing equivalent intelligence improvements. The transcript also raises failed training runs and a possible shortage of high-quality training data as concerns.

The host points to OpenAI's o1 and o3 reasoning models as examples of scaling compute at inference time. The longer o1 could think, the better it performed; the transcript says o3 later achieved strong results across software engineering, math, and science benchmarks. The proposed shift is to give models more compute for their chain of thought on harder problems, alongside or instead of scaling pretraining.

The host says scaling principles also appear relevant to image diffusion, protein folding, chemical models, and robotics world models. The episode presents language model scaling as potentially reaching a midgame while other modalities remain at an earlier stage.