The data black hole at the center of AI

Patel argues that recent gains come mainly from broader, better data rather than clear improvements in training sample efficiency. He describes reinforcement learning as synthetic data generation: compute searches for good solutions using verifiers or rubrics, then models learn from those examples. This process depends on human experts supplying trajectories, completions, and rubrics for specific skills.

Patel compares model training to years of instruction and practice, noting that GRPO can generate hundreds or thousands of rollouts for a task. He says frontier models train on tens to hundreds of trillions of tokens, while a person may encounter about 200 million language tokens by adulthood. He also points to robotics and autonomous driving: millions of demonstration hours still fall short of enabling complex open-ended robot tasks, and self-driving models use far more training data than a teenager's roughly 20 hours of driving practice.

Patel responds that the human genome is too small to contain a pretrained brain network, and suggests evolution may instead have shaped learning conditions such as hyperparameters and loss functions. He argues that pretrained AI models still need large datasets for each new skill, unlike people learning a programming language after education. He also says sensory input alone does not explain human intelligence, and cites scaling-law estimates suggesting infinitely more parameters would reduce required data by only about a factor of ten.

Patel says labs can profitably train models on common white-collar tasks because the skills can be taught at scale and shared across billions of sessions. Some jobs are predictable, while software engineering involves more out-of-distribution problems; he predicts demand for human software engineers could be higher in 2028 than now because AI complements their work. He frames automating AI research as the labs' route to addressing the remaining sample-efficiency problem, while leaving the implications for a future post.