François Chollet: How We Get To AGI

Chollet says falling compute costs and larger datasets helped deep learning make rapid progress, especially in language and vision. But after roughly 50,000-fold growth in pretrained model scale, GPT-4.5 reached only about 10% on ARC-1, which he designed to test solving unfamiliar problems. He argues that scaling pretraining alone did not produce fluid intelligence.

Chollet distinguishes task-specific skill from the ability to handle situations a system has not encountered before. He describes intelligence as using past information efficiently to meet future challenges, and says conventional exams often measure memorized knowledge or fixed skills instead. ARC-1 was designed to test whether models could solve novel tasks using basic concepts such as geometry, counting, and objectness.

Chollet credits test-time adaptation, where models change their behavior during inference, with progress on ARC. He says OpenAI’s ARC-fine-tuned o3 reached human-level performance on ARC-1, but that benchmark saturates and does not establish human-level intelligence. ARC-2 requires more deliberate reasoning: he reports baseline models at 0%, static reasoning systems around 1% to 2%, and test-time learning systems still well below people. ARC-3 is planned to assess interactive agency and efficiency under an action limit.

Chollet’s “kaleidoscope hypothesis” holds that seemingly novel situations can be understood by recombining a small set of reusable abstractions. He distinguishes continuous, value-based abstraction, which supports perception and pattern recognition, from discrete, program-based abstraction, which supports reasoning and planning. He proposes combining deep learning’s fast intuition with discrete program search, using learned guidance to make search more tractable.

Chollet describes a system that assembles models for new tasks by searching a growing library of reusable building blocks. The system would combine deep learning components for perception with algorithmic modules for reasoning, then add useful new abstractions back to its library. At AIndy, his research lab, the stated first milestone is solving ARC-AGI with a system that starts with no ARC-AGI-specific knowledge, with a longer-term aim of helping researchers accelerate scientific discovery.