Physicists have conducted a standardized evaluation of AI agents on research-level problems and obtained a result that the authors call "strong." In the TheoPhysEval test, consisting of 50 unpublished theoretical physics problems, OpenAI's GPT-6-Astra model operating within the Codex environment successfully solved 49 out of 50 problems. Moreover, in several instances, researchers initially dismissed the model's correct answers as mistakes—only to discover after consulting experts that the flaws lay in their own benchmark solutions. This turned an AI accuracy check into a quality audit of the control answers themselves.
What Was Tested: 50 Unpublished Problems Across 11 Categories
For the test, a team of physicists gathered 50 problems spanning 11 areas of theoretical physics. At the time of testing, none of the problems had been published anywhere, although each allowed for result verification. The authors intentionally excluded major unsolved problems: the goal was not to make the model produce a fundamentally new discovery, but to measure how well an agent performs the specific computational work researchers encounter. In many tasks, the initial values were modified compared to the researchers' recent papers, so the ready-made answer could not simply be copied from a publication—only the method itself remained applicable. According to the authors' estimates, a world-class expert typically spends nearly a day on such a problem, while a specialist from a related field takes significantly longer.
Results: 98% Accuracy at Under Three Dollars Per Problem
In the mode with access to the internet, scientific literature, and computational tools, Astra achieved a 98% accuracy rate, spending an average of less than $3 per problem. For comparison: Anthropic's Opus 5.5 in Claude Code solved 46 out of 50 problems (92%), Gemini 3.8 Flash solved 34, DeepSeek-V4.1-Flash solved 27, and GLM 5.3 solved 21. Thus, the gap between the leader and the rest of the test participants proved substantial, and the cost of solving a single research problem was comparable to the price of a cup of coffee.
When the AI Was Right and the Scientists Were Wrong
Several of Astra's answers were initially rejected as erroneous. However, after re-evaluation and consultations with experts, the authors discovered flaws in their own benchmark solutions; once corrected, the AI model's calculations were deemed correct in all such cases. The only problem that Astra failed to solve even with web search could not be solved by other models either: it required reconstructing a differential equation from a given curve, and the ambiguity of the condition significantly complicated finding a solution.
Memory Instead of Search
The models' accuracy changed very little when internet access was disabled: without web search, Astra and Opus both showed an identical result of 92%. According to the authors, this may indicate that leading models have essentially internalized a significant portion of scientific literature and are capable of reproducing the necessary formulas without searching through sources. At the same time, the authors emphasize that it is impossible to completely rule out the influence of training data based solely on this study.
The Limit of Capabilities: Calculation Is Not Yet Research
Success in calculations does not mean the AI is already capable of conducting full-scale physical research independently. Models are still noticeably worse at choosing a significant problem, formulating the right question, and knowing when to change the direction of work than they are at solving problems with a verifiable answer. TheoPhysEval, as intended by its authors, measures precisely the computational component of research labor—and in this narrow but important domain, AI agents demonstrated an expert-level performance.
