Chollet contrasts recurrent models, which update a hidden state repeatedly, with transformers that process input in parallel during training. He argues that fixed-depth models can run out of steps on tasks such as sorting, where comparisons require iterative computation. Chain of thought and tool use can help, but their effectiveness depends on training data or known algorithms.
Recursion Is The Next Scaling Law In AI
Speakers
François Chollet, visiting partner at Y Combinator, discusses recursive reasoning in AI with the Decoded host, Wa C.
The hierarchical reasoning model (HRM) repeatedly applies shared weights through low-level, high-level, and outer refinement loops. The speakers describe a 27-million-parameter model trained from scratch on roughly a thousand ARC tasks, reporting about 70% on ARC Prize 1 at the time. Its training uses stop-gradient and fixed-point-style updates rather than backpropagating through every recursive step; the speakers say the outer refinement loop was especially important.
The tiny recursive model (TRM) keeps the outer refinement approach while sharing weights in a single network and using a smaller architecture. It backpropagates through one full recursive step, while still truncating the longer recursion. The speakers report a 7-million-parameter model reaching 87% on ARC-1, compared with HRM’s reported 70%, and discuss task-specific results such as Sudoku and mazes.
Chollet argues that recursion and outer refinement are promising, though the mechanism behind their effectiveness remains unclear. The models discussed are task-specific, while large language models offer broader learned representations. The speakers suggest combining those representations with recursive reasoning in latent space to tackle more complex tasks.
Takeaways
- For tasks requiring many sequential operations, consider adding iterative computation instead of only increasing model size.
- Treat hidden-state updates as working memory that can accumulate intermediate information during reasoning.
- Test whether outer refinement improves your task; the speakers describe it as a key contributor in HRM and TRM.
- Compare model size and compute depth separately: a smaller model with recursion may outperform a larger fixed-depth model on a specific task.
- Keep task scope in mind: the reported recursive models were specialized, and their results do not establish general-purpose performance.
Mentioned in this talk
Search inside this talk
What was claimed 8
- featureHierarchical Reasoning Model: Uses low-level, high-level, and outer refinement recursion
“There are exactly three levels of recursion: the low level, the high level, and the outer refinement steps.”
YouTube Unverified1 source· Talk
- metricHierarchical Reasoning Model: 70% on ARC Prize 1
“This got something like 70% on ARC Prize 1.”
YouTube Unverified1 source· Talk
- featureHierarchical Reasoning Model: Uses fixed-point iteration with stop-gradient instead of backpropagating through all recursion steps
“What they do is use this DEQ method of fixed-point iteration.”
YouTube Unverified1 source· Talk
- featureTiny Recursive Model: Uses truncated backpropagation through one full recursion loop
“Rather than going through just one call to the H-net and the L-net, it goes through one full recursion loop.”
YouTube Unverified1 source· Talk
- featureTiny Recursive Model: Shares one network across low-level and high-level processing while keeping separate latent states
“You share weights between the L-net and the H-net, and it's just called the net.”
YouTube Unverified1 source· Talk
- featureTiny Recursive Model: Can learn to solve tasks without chain-of-thought supervision
“It's able to discover things without being teacher-forced via chain of thought.”
YouTube Unverified1 source· Talk
- metricTiny Recursive Model: 87% on ARC-1
“It actually gets from 70% to 87% on ARC-1.”
YouTube Unverified1 source· Talk
- metricTiny Recursive Model: 7 million parameters
“Now she brings it down to a 7-million-parameter model.”
YouTube Unverified1 source· Talk
