---
title: "AI Agents Can Calculate, But Don't Always Understand Science: An Experiment on Replicating Astronomy Papers"
description: "Researchers tested AI agents on reproducing 14 astronomy papers: in a control case, 12 variants yielded estimates from 2.16 to 3.53 kpc instead of the published 2.70, with only one matching."
date: 2026-10-18T11:00:21.686Z
lang: en
url: https://xab.info/en/posts/ai-agents-and-reproduction-of-astronomy-research
tags: [ai-agents, astronomy, reproducibility, llm, scientific-research]
author: "Alexander Buran"
publisher: "XAB.info"
---

# AI Agents Can Calculate, But Don't Always Understand Science: An Experiment on Replicating Astronomy Papers

![Draft scientific calculations, charts, and formulas on paper next to a monitor on a dark desk](https://xab.info/media/2026/10/18/ii-agenty-i-vosproizvedenie-astro-nomicheskikh-issledovanii/ii-agenty-i-vosproizvedenie-astro-nomicheskikh-issledovanii-1.webp)

## Key Points

- In the control experiment, 12 analysis variants of a single dataset yielded estimates from 2.16 to 3.53 kiloparsecs versus the published ~2.70; only one variant matched.
- A key parallax correction (+0.02 mas) was already specified in the paper, but AI agents did not link it to the result prior to performing the analysis.
- The authors propose evaluating scientific AIs across the entire reproduction process rather than just matching the final number.

Modern language models are capable of extracting parameters from scientific publications and performing numerical calculations, but this is not always enough to independently replicate a study from start to end. In the paper "Reconstructing Implicit Scientific Knowledge: Evaluating LLM Agents through End-to-End Reproduction of Astronomy," the authors tested how well AI agents can reproduce astronomical results based solely on a single article. In a control experiment, twelve predetermined analysis options for the exact same dataset yielded spatial scale estimates ranging from 2.16 to 3.53 kiloparsecs, whereas the original publication reported a value of about 2.70 kiloparsecs. A match with the published figure was achieved using only one of the twelve methods, and the key correction without which replication was impossible was already explicitly stated in the paper itself.

### How the Experiment Was Designed

The authors tested the approach on fourteen astronomical studies: one published in The Astrophysical Journal and thirteen articles from Nature. In eleven of the latter thirteen, they identified methodological or interpretational ambiguities that made it impossible to unambiguously determine the sequence of steps needed to reproduce the analysis. In the control case, the researchers predetermined twelve calculation variants combining various object selection methods, sky region masking, and parallax zero-point corrections. Recall that parallax allows estimating the distance to a celestial body based on the apparent shift in its position against more distant background objects. Differences in settings led to a spread of estimates from 2.16 to 3.53 kiloparsecs instead of the published ~2.70.

### Why Only One Value Matched

Crucially, the authors emphasize: the original result was not used for parameter tuning or stopping calculations. The matching variant was discovered only after all twelve analyses were fully completed. This rules out trivial "chasing" of the number and shows that the problem is not about guessing the answer, but about choosing the correct methodology. It is also telling that the discrepancy was not due to missing data: the +0.02 milliarcsecond parallax correction was already contained in the article, yet the AI agents did not recognize its causal link to the result until the analysis itself demonstrated the impact of this correction.

### Implicit Knowledge as a Bottleneck

The researchers categorize the causes of failed reproductions into several types: computational error, insufficiently defined methodology, and the inability to link disparate information from the article into a coherent analysis logic. From this follows an important conclusion: a simple match with the published figure does not yet prove that the system has reconstructed the authors' reasoning path—the desired result could be obtained by chance or through one of several acceptable processing variants. Symmetrically, a mismatch does not necessarily mean that the original scientific work is flawed. The problem essentially lies in understanding why a specific parameter must be applied to these exact data and within this specific study, rather than in the mere ability to find a number.

### What This Means for Evaluating Scientific AIs

On this basis, the authors propose evaluating scientific AI systems across the entire reproduction process—from data and settings selection to result verification—rather than just checking whether the final number matches the publication. This approach helps identify decisions that remain implicit in the paper because experts make them based on experience and professional conventions. At the same time, the researchers note that even an audit of computations does not replace domain knowledge: to understand which assumptions are justified and which change the meaning of the result, the system needs to restore the connection between the methodology and the scientific task. The findings indicate that the next step in the evolution of scientific AI agents is not increasing arithmetic precision, but developing the ability to interpret implicit methodological knowledge.

## Fact-Check & Sources

- [ИИ может найти нужные данные в научной статье, выполнить расчёты и всё равно не понять, как именно авторы пришли к результату](https://www.ixbt.com/news/2026/10/18/ii-mozet-naiti-nuznye-dannye-v-naucnoi-state-vypolnit-rascety-i-vse-ravno-ne-poniat-kak-imenno-avtory-prisli-k-rezultatu.html) - The primary source contains all key facts: paper title, 14 studies (1 ApJ + 13 Nature), 11/13 with ambiguities, 12 variants, range 2.16–3.53 kpc vs ~2.70, +0.02 mas correction, methodology without result-fitting, and the recommendation to evaluate the entire reproduction process.

## FAQ

### How many analysis variants matched the published result?

Out of twelve predetermined variants, only one matched the published value of about 2.70 kiloparsecs; the rest yielded estimates ranging from 2.16 to 3.53 kiloparsecs.

### Did the authors use the original result to tune the calculations?

No. The authors emphasize that the original result was not used for parameter tuning or stopping calculations; the matching variant was found only after all twelve analyses were completed.

### Why didn't the AI agents reproduce the result if the necessary correction was in the paper?

The +0.02 milliarcsecond parallax correction was indeed stated in the paper, but the agents did not recognize its causal link to the result until the analysis showed its impact. The issue was understanding why the parameter needed to be applied to these specific data.