---
title: "GPT-6-Astra Solves 49 Out of 50 Theoretical Physics Problems and Finds Errors in Scientists' Own Solutions"
description: "Physicists tested AI agents on 50 unpublished theoretical physics problems: GPT-6-Astra solved 49 out of 50, and in several cases, researchers identified errors in their own benchmark solutions."
date: 2026-10-18T11:08:23.245Z
lang: en
url: https://xab.info/en/posts/gpt-6-astra-solves-49-physics-problems
tags: [gpt-6-astra, ai-agents, theoretical-physics, theo-phys-eval, openai]
author: "Alexander Buran"
publisher: "XAB.info"
---

# GPT-6-Astra Solves 49 Out of 50 Theoretical Physics Problems and Finds Errors in Scientists' Own Solutions

![Handwritten complex mathematical formulas and graphs on paper next to a tablet on a desk](https://xab.info/media/2026/10/18/gpt-6-astra-reshila-49-iz-50-zadach-po-fizike/gpt-6-astra-reshila-49-iz-50-zadach-po-fizike-1.webp)

## Key Points

- In the TheoPhysEval test of 50 unpublished problems, GPT-6-Astra solved 49, achieving 98% accuracy at a cost of under $3 per problem.
- In several instances, researchers initially mistook the model's correct answers for errors, but upon re-evaluation found flaws in their own benchmark solutions.
- Accuracy dropped very little without internet access, which authors attribute to the model memorizing a large portion of scientific literature.

Physicists have conducted a standardized evaluation of AI agents on research-level problems and obtained a result that the authors call "strong." In the TheoPhysEval test, consisting of 50 unpublished theoretical physics problems, OpenAI's GPT-6-Astra model operating within the Codex environment successfully solved 49 out of 50 problems. Moreover, in several instances, researchers initially dismissed the model's correct answers as mistakes—only to discover after consulting experts that the flaws lay in their own benchmark solutions. This turned an AI accuracy check into a quality audit of the control answers themselves.

### What Was Tested: 50 Unpublished Problems Across 11 Categories

For the test, a team of physicists gathered 50 problems spanning 11 areas of theoretical physics. At the time of testing, none of the problems had been published anywhere, although each allowed for result verification. The authors intentionally excluded major unsolved problems: the goal was not to make the model produce a fundamentally new discovery, but to measure how well an agent performs the specific computational work researchers encounter. In many tasks, the initial values were modified compared to the researchers' recent papers, so the ready-made answer could not simply be copied from a publication—only the method itself remained applicable. According to the authors' estimates, a world-class expert typically spends nearly a day on such a problem, while a specialist from a related field takes significantly longer.

### Results: 98% Accuracy at Under Three Dollars Per Problem

In the mode with access to the internet, scientific literature, and computational tools, Astra achieved a 98% accuracy rate, spending an average of less than $3 per problem. For comparison: Anthropic's Opus 5.5 in Claude Code solved 46 out of 50 problems (92%), Gemini 3.8 Flash solved 34, DeepSeek-V4.1-Flash solved 27, and GLM 5.3 solved 21. Thus, the gap between the leader and the rest of the test participants proved substantial, and the cost of solving a single research problem was comparable to the price of a cup of coffee.

### When the AI Was Right and the Scientists Were Wrong

Several of Astra's answers were initially rejected as erroneous. However, after re-evaluation and consultations with experts, the authors discovered flaws in their own benchmark solutions; once corrected, the AI model's calculations were deemed correct in all such cases. The only problem that Astra failed to solve even with web search could not be solved by other models either: it required reconstructing a differential equation from a given curve, and the ambiguity of the condition significantly complicated finding a solution.

### Memory Instead of Search

The models' accuracy changed very little when internet access was disabled: without web search, Astra and Opus both showed an identical result of 92%. According to the authors, this may indicate that leading models have essentially internalized a significant portion of scientific literature and are capable of reproducing the necessary formulas without searching through sources. At the same time, the authors emphasize that it is impossible to completely rule out the influence of training data based solely on this study.

### The Limit of Capabilities: Calculation Is Not Yet Research

Success in calculations does not mean the AI is already capable of conducting full-scale physical research independently. Models are still noticeably worse at choosing a significant problem, formulating the right question, and knowing when to change the direction of work than they are at solving problems with a verifiable answer. TheoPhysEval, as intended by its authors, measures precisely the computational component of research labor—and in this narrow but important domain, AI agents demonstrated an expert-level performance.

## Fact-Check & Sources

- [GPT-6-Astra решила 49 из 50 задач по теоретической физике — и нашла ошибки в расчётах учёных](https://www.ixbt.com/news/2026/10/17/gpt-6-astra-resila-49-iz-50-zadac-po-teoreticeskoi-fizike-i-nasla-osibki-v-rascetax-ucenyx.html) - Primary source; contains all key figures: 49/50, 98%, <$3, Opus 5.5 results (46), Gemini 3.8 Flash (34), DeepSeek-V4.1-Flash (27), GLM 5.3 (21), benchmark self-correction, and the single unsolved problem.
- [GPT-6-Astra решила 49 задач по физике и нашла ошибки в ответах учёных](https://itzine.ru/news/ai/gpt-6-astra-49-zadach-fizika-oshibki-uchenyh.html) - Confirms the name TheoPhysEval, 50 problems from 11 categories, modification of initial values, and expert time estimates.
- [GPT-6-Astra решила 49 из 50 задач в тесте по теоретической физике](https://cisoclub.ru/gpt-6-astra-reshila-49-iz-50-zadach-v-teste-po-teoreticheskoj-fizike/) - Confirms OpenAI's Codex environment, <$3 per problem, 11 categories, 92% for Opus 5.5, and identical Astra/Opus results without internet.
- [GPT-6 Astra: OpenAI хочет превратить ИИ из помощника в исполнителя задач](https://habr.com/ru/companies/otus/news/1078608/) - Contextual source regarding OpenAI's stance on the model; does not contradict test figures.

## FAQ

### What is the TheoPhysEval test?

It is a set of 50 unpublished theoretical physics problems across 11 categories, compiled by a team of physicists to evaluate AI agents. The tasks test the application of known methods to modified conditions rather than making new discoveries; they were unpublished at the time of testing.

### How many problems did GPT-6-Astra solve and how much did it cost?

Astra solved 49 out of 50 problems (98% accuracy) in the mode with access to the internet and tools, spending an average of less than $3 per problem within OpenAI's Codex environment.

### Why did the authors initially consider the model's answers wrong?

In several cases, researchers mistook Astra's correct answers for mistakes, but after expert consultations discovered flaws in their own benchmark solutions; after corrections, the AI's calculations were recognized as correct.

### Does this mean AI is conducting full-scale physical research?

No. The authors emphasize that success in calculation does not equal the ability to independently conduct research: models are still worse at choosing significant problems, formulating questions, and changing work direction than at solving problems with a verifiable answer.