This is the State of the Art in Robotics

Finn argues that useful physical AI must act autonomously because its decisions directly affect the world. Unlike recommendation systems, robots need to make far fewer mistakes. She points to Waymo’s quarter-million weekly rides as evidence that trustworthy autonomy in the physical world is possible.

A robot making espresso must handle precise force, liquid, and timing, with a target above 90% reliability. Finn’s team improves policies through repeated experience, using human intervention to recover from dead ends and a general value model to assess progress. For espresso, the robot exceeded 90% success; reinforcement learning roughly doubled throughput on evaluated tasks.

Most robotics foundation models act on current sensor observations without memory, which makes long sequences difficult. Finn describes combining about 10 seconds of efficient video memory with longer textual summaries. This enabled a robot to clean a kitchen autonomously for 10 to 15 minutes, tracking varied steps such as wiping counters and putting dishes away.

Finn’s team trained the π0.7 model on diverse robot demonstrations, rollout data, human video, and web data, with prompts that include task context and data quality. The single pretrained model matched or outperformed fine-tuned specialists on coffee and box tasks. It also showed compositional generalization, including folding clothes on a different robot platform without folding data from that platform.

Finn says robotics has moved toward generalist models that work out of the box, with PI models adapted for laundry, warehouse packaging, and other embodiments. In the Q&A, she says robotics will need large-scale real robot experience alongside human video and web data. She expects open-source models to support a strong ecosystem, while noting that physical hardware and data make distribution slower than for language models.