A study by Nvidia, whose results the company shared with TechCrunch, shifts the center of gravity in AI agent development: according to the engineers' conclusions, it is the software shell, not the base language model, that determines success when solving tasks with a long planning horizon. The key demonstration was the work with the Claude Opus 5 model on the ARC-AGI-3 benchmark — a set of 2D games with no instructions, where the system must independently understand the rules and learn to win, much like a human.
The Shell Decides: How Claude Opus 5 Scored 100% on ARC-AGI-3
Thanks to a special software shell for efficient memory management, augmented by a component resembling a "supervisor," researchers achieved a 100% score for Claude Opus 5 on the interactive ARC-AGI-3 test. Without this shell, the same model scored only 30% — and, according to Nvidia, this was the best result among all tested models. Thus, the gap between the "bare" model and the model within an agent infrastructure turned out to be several-fold.
"An Agent Is Not a Model's API": Nvidia's Position
"Overall, the world perceives an agent almost as a model's API," Adel El Hallak, Vice President of Products in Nvidia's AI division, told TechCrunch. In his words, an agent today is something more: "It's the model. It's the structure around the model that we call scaffolding, i.e., the set of tools you use. It's the execution environment, as well as the associated skills and libraries that we give you access to." Choosing a model, as Nvidia emphasizes, is not enough — it is precisely the management system that processes memory, context, and feedback that turns a model into an agent.
The Supervisor as the "CEO" of AI
A separate emphasis in the study is placed on introducing a supervisory agent in addition to the main agent performing the work. According to El Hallak, it "acts almost like a CEO, nudging the agent when it strays from course or starts exploring a path that may lead to a dead end, or re-explores a path it has already taken." It is precisely this two-tier structure, in Nvidia's view, that allows maintaining focus on the long-term task without getting stuck in loops or losing context.
Contradictory Data
It is appropriate here to compare two perspectives. On the one hand, Nvidia demonstrates that it is precisely a complex agent shell with a supervisor that brought Claude Opus 5 to 100% on ARC-AGI-3, while without it the result dropped to 30%, with OpenAI models scoring less than 10% on the same test. On the other hand, OpenAI last month conducted its own study and found that simply replacing two parameters in the test environment tripled the models' performance. This raises the question of how much the shell architecture, as opposed to fine-tuning the environment and parameters, is the determining factor: Nvidia's interpretation emphasizes the role of agent infrastructure, while OpenAI's data point to the high sensitivity of the result to the test environment conditions. Both sides agree that the "bare" model is only part of the system, but they differ in assessing the contribution of the shell versus the environment parameters.
What This Means for Long-Term AI Tasks
The results confirm: while model choice matters, it is a much smaller part of the agent system than is generally thought, especially for long-term tasks. Figuring out how to make AI perform tasks with a long planning horizon without getting distracted or stuck in one place remains one of the main goals of research in AI agents. The practical takeaway for developers — investments in memory, context, tools, and supervisory logic can yield greater gains than swapping the base model.
