Chinchilla

The talk presents three perspectives on data as a central bottleneck in building capable AI systems. The host argues that production performance depends on inspecting real-world failures and investing in expert data and environments, rather than focusing only on model architecture. Vincent Chen describes how Snorkel scales expert supervision, from data programming to validation agents and benchmarks that assess coding quality. Volo presents Inception Labs’ diffusion language models and Tau Forge for creating realistic training and evaluation environments, while Shane explains how multilingual data transfer can guide language mixtures, model sizing, and pretraining choices.

The talk explains how early scaling laws showed that increasing model size, training data, and compute reliably improved AI performance. OpenAI’s research and DeepMind’s Chinchilla results established that models also need enough training data to use their capacity effectively. It then considers whether pretraining gains are slowing and presents test-time compute, where models spend more compute reasoning through a problem, as a new scaling direction. The closing argument is that scaling may still have substantial room to grow in reasoning models and other areas such as image generation, protein folding, and robotics.