next-token prediction

Nick Joseph describes how Anthropic approaches pretraining: predicting the next token across large datasets and using compute to improve model quality. He explains the engineering challenges of training at scale, from parallelizing workloads and diagnosing hardware failures to coordinating specialists across teams. He discusses how pretraining relates to reinforcement learning, post-training, alignment, and synthetic data, emphasizing that many choices need empirical testing. He also argues that inference efficiency and subtle engineering bugs will shape future progress, and that students entering AI should focus on engineering and how to make AGI beneficial.