Jeff Dean’s Lecture for YC AI

Dean says neural networks were promising in the 1980s and 1990s, but limited data and compute made other approaches more practical. He recalls estimating that parallel training needed about 100,000 times more compute, which has since become available. Google Brain explored large-scale learning and built tools to shorten the time between experiments.

Dean describes TensorFlow as an open-source platform designed to support flexible research and deployment across data centers, phones, CPUs, GPUs, and TPUs. He says it was built to address issues in Google’s earlier internal system and to bring researchers and external contributors into a shared project. He also emphasizes reproducible performance benchmarks and faster experimental turnaround.

Dean illustrates how similar model structures can serve different tasks when trained on different data, including finding text in Street View images, identifying rooftops for solar estimates, and assessing diabetic retinopathy. He describes a retinal model evaluated against ophthalmologists and clinical trials in India. In quantum chemistry, a neural network trained on simulator results reportedly matched its accuracy while running 300,000 times faster.

Dean explains sequence-to-sequence models through Gmail Smart Reply and Google Translate, where neural translation improved human-rated quality over phrase-based systems with far less code. He then describes architecture search and learned optimizer rules that use reinforcement learning to test candidate designs. One search ran 12,000 experiments over a weekend, while learned architectures performed well on image and language tasks.

Dean says deep learning can use reduced-precision arithmetic, motivating Google’s Tensor Processing Units. He describes a TPU pod with 256 chips and 11.5 petaflops, programmable through TensorFlow, and says Cloud TPU access was planned. In questions, he argues that models trained across many tasks could improve data efficiency and support more flexible behavior, and identifies healthcare as a promising application while noting privacy and rollout challenges.