Chelsea Finn: Building Robots That Can Do Anything

Finn says robotics applications often require separate companies to build custom hardware, software, and movements for each setting. Physical Intelligence is pursuing a model that can help different robots perform different tasks. She argues that large-scale data is necessary, but industrial, video, and simulation data each have limits in task diversity, learning value, or realism.

Finn describes training a robot to fold laundry, beginning with single shirts and gradually adding crumpled clothing and varied items. After months of poor results, the team improved performance by pre-training on broad robot data and fine-tuning on curated demonstrations. A 3-billion-parameter vision-language model further improved speed and folding consistency: the robot folded five items in nine minutes and handled some clothing it had not seen during fine-tuning.

To test whether robots could work in unfamiliar places, the team collected tidying data across more than 100 rooms and combined it with other robot and instructional data. The mobile-manipulation data made up only 2.4% of the pre-training mixture. In three unfamiliar Airbnbs, the robot tidied kitchens and bedrooms; tests also showed that broader environmental diversity improved performance, though success was around 80% and failures remained.

Finn describes a hierarchical system that turns user requests into verbal subtasks for a lower-level robot policy. The team used language models to generate hypothetical prompts from existing robot demonstrations, avoiding the need to collect every interaction directly. The system handled requests such as making a sandwich without pickles and corrections to clean up only the trash, while existing foundation models performed worse in the team's instruction-following evaluation.

Finn says real-world robot data is important but not sufficient, and that general-purpose robots still need more research before they can reliably handle open-world tasks. In the Q&A, she points to reinforcement learning as a way to improve success and speed, and discusses the need for fast on-robot inference and infrastructure for multimodal data. She also says simulation may help with evaluation, while online robot experience can support post-training.