Robot-Use Agents: Why General-Purpose Models May Win in Robotics

Jay describes RT-2, which fine-tuned a language model pretrained on web text and images to output robot end-effector poses. The discussion contrasts that approach with newer models that can use tools, write code, and adapt in context. The guests suggest that broad pretraining data may matter as much as the model architecture.

The guests trace coding agents from Voyager, which created reusable tools for Minecraft, to robotics research that used Python functions such as picking up an object or moving to a pose. They say models could combine these functions into complex tasks without additional robot data. François outlines limits of in-context learning: performance can fluctuate and often saturates after roughly 20 to 40 examples.

Astra controls robot arms using camera feeds and commands to move a block into a bowl. The guests distinguish direct, repeated model calls from a harness that can run compiled skills quickly and use a vision-language model at decision points. They identify model latency as a bottleneck and discuss consolidating successful attempts into reusable policies.

The panel discusses the idea that capable models trained on different data may learn similar representations of the world. One guest suggests that vision, CAD, and computer-use data can teach spatial concepts such as orientation and movement. The panel proposes that combining these data types with robotics tools could improve robot control.

A guest says some frontier labs and robotics companies expect general-purpose robots within about two years, describing systems that could follow natural-language instructions across unfamiliar tasks and settings. The panel points to latency and economic usefulness as challenges. They discuss collecting robot experiences, organizing skills, and distilling what works into faster models.