How Intelligent Is AI, Really?

Kamradt describes intelligence using François Chollet’s definition: the ability to learn new things. ARC-AGI tests that ability with tasks that ordinary people can solve, rather than increasingly difficult questions aimed at expert-level knowledge. He contrasts this with AI systems that excel at specific skills such as chess or Go but may not transfer those skills to new problems.

Kamradt recalls that GPT-4 without reasoning scored about 4% or 5% on ARC, while early tests of o1 reached about 21%. He says the jump helped show how much reasoning methods could change performance. Major labs including OpenAI and xAI have since reported ARC-AGI results, though he cautions that their adoption is secondary to the foundation’s goal of advancing open research.

Kamradt warns that reinforcement learning environments can produce strong benchmark scores without showing that a system can handle future problems. Since it is impractical to build an environment for every possible task, he argues that research should focus on systems that generalize without task-specific training. Hidden test sets help assess performance on problems the system has not seen.

Kamradt describes ARC-AGI-1 as a static benchmark of 800 tasks created by François Chollet, and ARC-AGI-2 as a deeper version released in March 2025. He says ARC-AGI-3 is planned as an interactive benchmark with about 150 game-like environments and no instructions: systems must act, observe responses, and infer the goal. Each environment will be checked with ordinary people, and those that do not meet a solvability threshold will be excluded.

Kamradt says intelligence evaluation should consider training data and energy as well as accuracy; wall-clock time can change with added compute. ARC-AGI-3 will compare how many actions AI systems and humans need to complete each environment, limiting brute-force play. He says a perfect ARC score would be strong evidence of generalization, but not sufficient to establish AGI.