WhoBuilt

How Scaling Laws Will Determine AI's Future

Jan 23, 2025· 10 min· 41K views·Watch on YouTube
#Model Training#Language Models#Training Data#Reasoning Models#Multimodal Models

An unnamed Y Combinator host explains how scaling laws shaped large language models and how test-time compute may create a new path for AI progress.

The early case for scaling

The host describes the shift from GPT-2, with 1.2 billion parameters, to GPT-3, which was more than 100 times larger. A 2020 OpenAI paper by Jared Kaplan, Sam McCandlish, and colleagues found that increasing model size, training data, and compute brought steady performance gains. The host says those gains followed a power law and depended more on scale than on algorithm changes.

Chinchilla adds a data constraint

Google DeepMind's 2022 research examined how to allocate a compute budget between model size and training data. Its researchers trained more than 400 models and concluded that models such as GPT-3 were undertrained. Chinchilla, less than half GPT-3's size but trained on four times as much data, outperformed models more than twice its size at one point.

Questions about pretraining's limits

The host describes debate over whether larger, more expensive models are still delivering comparable capability gains. One unnamed participant says companies are increasing GPU use without seeing equivalent intelligence improvements. The transcript also raises failed training runs and a possible shortage of high-quality training data as concerns.

Test-time compute as another scaling path

The host points to OpenAI's o1 and o3 reasoning models as examples of scaling compute at inference time. The longer o1 could think, the better it performed; the transcript says o3 later achieved strong results across software engineering, math, and science benchmarks. The proposed shift is to give models more compute for their chain of thought on harder problems, alongside or instead of scaling pretraining.

Scaling beyond language models

The host says scaling principles also appear relevant to image diffusion, protein folding, chemical models, and robotics world models. The episode presents language model scaling as potentially reaching a midgame while other modalities remain at an earlier stage.

Takeaways

  • When planning model training, balance parameter count, training data, and available compute.
  • Compare models by training data as well as size; Chinchilla showed that a smaller, better-trained model could outperform larger ones.
  • Track capability gains against GPU and compute growth to spot diminishing returns.
  • Evaluate test-time compute as a way to improve performance on harder tasks by letting models reason longer.
  • Look for scaling opportunities beyond text, including image, science, and robotics models.

Mentioned in this talk

OpenAIGoogle DeepMindGPT-3Scaling Laws for Neural Language ModelsClaude SonnetGPT-4oo1o3GPT-2Chinchillaimage diffusion modelsprotein folding modelsWorld modelsScaling lawstest-time scalingJared KaplanSam McCandlishGwern

Search inside this talk

What was claimed 15

Related talks