Scaling laws

Nick Joseph describes how Anthropic approaches pretraining: predicting the next token across large datasets and using compute to improve model quality. He explains the engineering challenges of training at scale, from parallelizing workloads and diagnosing hardware failures to coordinating specialists across teams. He discusses how pretraining relates to reinforcement learning, post-training, alignment, and synthetic data, emphasizing that many choices need empirical testing. He also argues that inference efficiency and subtle engineering bugs will shape future progress, and that students entering AI should focus on engineering and how to make AGI beneficial.

Tom Brown traces his path from early YC startups through OpenAI to co-founding Anthropic, describing how he learned to take ownership and moved into AI research. He explains how scaling laws shaped the work that led to GPT-3, and how Anthropic built training infrastructure before finding product-market fit with Claude 3.5 and coding. Claude Code began as an internal engineering tool, with a focus on giving Claude the tools and context to work effectively. He also discusses compute constraints, the trade-offs of using multiple chip platforms, and opportunities to help AI models do useful work across businesses.

Jared Kaplan explains how pre-training and reinforcement learning, scaled with more compute and data, have produced predictable improvements in AI capabilities. He describes scaling laws for training and task length, and argues that longer-horizon work will depend on organizational knowledge, memory, and better oversight. He expects AI to take on increasingly complex tasks, while human-AI collaboration remains important for work that needs judgment. He advises builders to experiment at the limits of current models and to find ways to integrate AI into more parts of the economy.

Sam Altman describes how OpenAI began with a small group committed to the unlikely goal of building AGI, and how scaling laws helped turn that research effort into products. He argues that falling model costs and new capabilities leave a large gap for startups to fill with new products, agents, and interfaces. He sketches a future of personalized AI that remembers, connects to data, acts proactively, and eventually works alongside robots. His advice to founders is to choose a meaningful problem, build what others overlook, hire for drive and accomplishment, and keep going through setbacks.

Bob McGrew traces OpenAI’s shift from early robotics and game projects toward large language models, arguing that scale and generalization connected those efforts. He describes how reasoning and test-time compute may help models overcome pretraining data limits and make agents reliable enough to take actions. He also discusses how AI startups should begin with the strongest available model, then optimize for cost after finding value with users. The talk closes on AI’s effects on work, the need for useful software tailored to people’s workflows, and the prospect of robotics and scientific progress.