Anthropic Head of Pretraining on Scaling Laws, Compute, and the Future of AI

Joseph describes pretraining as learning to predict the next word across large text datasets, producing a training signal at every token. He says scaling compute, data, and model size has reliably reduced prediction loss. Autoregressive models also have a practical advantage: they can generate open-ended text that can be used in products.

Choosing model dimensions and parallelism means balancing hundreds of hyperparameters, so teams test scaled-down versions before committing to expensive runs. In Anthropic’s early work, the team built and modified distributed training tools and profiled jobs across hundreds or thousands of GPUs. Joseph says hardware can fail in unexpected ways, and a subtle bug can waste months of training.

Joseph says it is unclear how much useful internet data remains, since usefulness depends on the model and cannot be counted easily. Training on a model’s own outputs may reproduce its errors, though AI-generated material online is not automatically harmful. He favors evaluations that measure meaningful capabilities, produce low-noise results, and run quickly; he also says next-token loss can be a useful signal.

Joseph defines alignment as making models share human goals and behave as intended. He sees post-training as easier to iterate on because teams can test changes in hours or days, while pretraining runs take months. Some alignment methods might eventually move into pretraining for greater robustness, but doing so can make later behavior changes harder.

Joseph expects further shifts in training methods, including more reinforcement learning, while saying scale and careful experimentation remain important. He emphasizes engineering depth, from model behavior down to GPUs and network protocols, and close coordination with inference teams so models can run efficiently. For students, he recommends building engineering skills and thinking about how to use AGI well.