Why The Harness Matters More Than The Model

The host argues that harnesses deserve research attention because changes to the scaffolding can substantially affect results. They cite an 18% improvement between two harnesses and describe ARC-AGI results rising from 30% to 95% with a harness, with Nvidia’s AVO reaching 100%. They also frame harnesses as a way to learn from test-time experience beyond what in-context examples alone can achieve.

The host traces harnesses from basic sampling and few-shot prompts through chain of thought, tools, memory, skills, reflection, and multi-agent systems. They call these static harnesses because the system prompt and harness do not improve themselves. Newer approaches such as DSPy optimize prompts, while Darwin Machines and continual harnesses can modify harness code and other system components.

Seth of Prime Intellect describes Prime Agent as a layer that gives language models persistent state, tools, compute, and coordinated subagents. Its design includes a Python REPL, context compaction, persistent subsessions, and ways to refine memory, skills, and prompts. Seth reports 95.5% on ARC-AGI with Opus and describes a seven-day Factorio run using 633 agents and 23 million output tokens.

John presents OpenJarvis, a personal AI stack designed to run inference, agents, memory, and learning on users’ devices. The project combines interfaces, agent logic, models and inference engines, tools, and optimization. John says cloud models can optimize local configurations, reporting lower costs and latency and up to 800 times lower cost for local inference.

Josh and Rean describe QM, YC’s open-source assistant for Slack and the web, with personal context, files, scheduled jobs, and shared channel use. They moved agent conversations into Postgres and treat sandboxes as resources agents can select, while keeping database writes subject to human review. They also discuss agents giving up too early and the challenges of preserving social context and permissions.