The host argues that robotics has repeatedly been declared nearly solved, from AlphaGo and MuJoCo to teleoperation datasets and vision-language-action models. They identify four stubborn challenges: inaccurate physical world models, difficult action representations, limited tactile sensing, and embodiment drift as hardware changes. Collecting teleoperation data is also demanding, making it hard to scale.
Why Robotics Still Isn't Solved - But Could Be Soon
Speakers
Y Combinator hosts a robotics research and startup discussion with Stanford researchers Marcel, Milan Ganai, and Tyler Lum, plus Rerun CEO Nico and General Intuition founders Bill and Guanming, on what still limits robotics and how to make progress.
Marcel presents Multi-Scale Embodied Memory, developed during an internship at Physical Intelligence, to help policies handle tasks longer than short demonstrations such as folding clothes or making a sandwich. Dense visual memory supports dexterous actions, while a high-level policy stores a compressed text summary of recent events. In examples, memory helps robots stop cooking at the right time and change strategy after repeated mistakes.
Milan Ganai describes R&B Encore, a self-supervised process for finding useful reasoning traces for different robot forms and tasks. A proposer generates candidate explanations, and a validator scores them for brevity, generalization, and whether they help predict actions. The reported results favor movement and gripper-position reasoning for manipulation, and structural affordances for legged navigation; reasoning can also be dropped at inference to avoid added latency.
Tyler Lum presents Sim-to-Real, which uses massively parallel reinforcement learning in simulation to train a single policy for a dexterous hand and arm. The policy tracks object poses from a human video, allowing it to handle unfamiliar tools and tasks without retraining; tests covered 12 unseen tool and behavior combinations. Pose tracking caused about 60% of failures, while random disturbances during training helped the robot recover from dropped objects.
Nico argues that robotics startups can begin with a paying customer problem, off-the-shelf hardware, and teleoperation, then use deployment experience to guide data collection and model improvement. He emphasizes customer-site replicas, repeatable evaluation, failure tracking, and data tools suited to multimodal, multi-rate physical data. Bill and Guanming of General Intuition then describe optimizing world action models by reducing diffusion steps and avoiding full future-frame decoding, reporting inference at 500 milliseconds per 16-action chunk on Jetson.
Takeaways
- Choose a customer problem that can work with teleoperation, then learn its real requirements through deployment.
- Train and evaluate on collected data early so failures can improve both the model and data collection.
- For long tasks, give policies a way to track progress and recover from earlier mistakes.
- Select reasoning signals for the embodiment and task; more annotations are not automatically more useful.
- Simulation can scale dexterity training, but pose tracking and real-world contact remain important bottlenecks.
Mentioned in this talk
Search inside this talk
What was claimed 17
- builtdata catalog and large-scale data backend
“We also have an infrastructure product called Rerun Hub”
YouTube Verified2 sources· Talk, Job posting· archived
- builtRerun built Parloa AI SDKopen-source SDK for physical data
“This open-source SDK is pretty popular”
YouTube Verified2 sources· Talk, Job posting· archived
- role atMilan Ganai is researcher at
Waymo“is currently working at Waymo”
YouTube Unverified1 source· Talk
- built
Physical Intelligence built memMulti-Scale Embodied Memory system“We call this system MEM, or Multi-Scale Embodied Memory.”
YouTube Unverified1 source· Talk
- previously atMarcel previously atPhysical Intelligenceintern
“some of the work I did during my internship at Physical Intelligence”
YouTube Unverified1 source· Talk
- featuremem: Combines dense short-context visual memory for a low-level policy with compressed long-context language memory for a high-level policy.
“Short-context memory consists of dense frames ... Long-context memory is a compressed language representation”
YouTube Unverified1 source· Talk
- featureR&B Encore: Proposes and scores reasoning traces for embodiment alignment, concision, non-triviality, and action predictiveness.
“a reasoning proposer ... a reasoning validator”
YouTube Unverified1 source· Talk
- metricSim2Real: Policy runs at 60 hertz controlling a 22-degree-of-freedom hand and 7-degree-of-freedom arm.
“It runs at 60 hertz and simultaneously controls both the 22-degree-of-freedom hand and the 7-degree-of-freedom arm.”
YouTube Unverified1 source· Talk
- featureSim2Real: Uses a human video to specify goal poses for a frozen policy; the video provides no robot actions and is not used to train or fine-tune the policy.
“The human video doesn’t provide robot actions and isn’t used to train or fine-tune the policy.”
YouTube Unverified1 source· Talk
- featurePlay to Perfect: Pretrains a shared dexterous prior through task-agnostic play, then fine-tunes it for sparse-reward contact-rich assembly.
“We first learn a shared dexterous prior through task-agnostic play”
YouTube Unverified1 source· Talk
- featureRerun Hub: Provides a data catalog and large-scale cloud data backend.
“a data catalog and large-scale data backend”
YouTube Unverified1 source· Talk
- featureParloa AI SDK: Supports logging, visualization, general-purpose querying, and data loading for training.
“It supports logging, visualization, general-purpose querying, and data loading for training”
YouTube Unverified1 source· Talk
- ceo ofCEO
“one of the co-founders and the CEO of Rerun”
YouTube Unverified1 source· Talk
- founded
“I’m Nico, one of the co-founders and the CEO of Rerun.”
YouTube Unverified1 source· Talk
- uses
“flow matching allows it to act on those action chunks”
YouTube Unverified1 source· Talk
- uses
“has a central diffusion transformer that tries to predict what the future looks like”
YouTube Unverified1 source· Talk
- metricGeneral Intuition: World action model runs at 500 milliseconds per chunk for 16 actions on Jetson.
“the world action model can run 500 milliseconds per chunk, which is 16 actions on the Jetson”
YouTube Unverified1 source· Talk
