Modern language models are capable of extracting parameters from scientific publications and performing numerical calculations, but this is not always enough to independently replicate a study from start to end. In the paper "Reconstructing Implicit Scientific Knowledge: Evaluating LLM Agents through End-to-End Reproduction of Astronomy," the authors tested how well AI agents can reproduce astronomical results based solely on a single article. In a control experiment, twelve predetermined analysis options for the exact same dataset yielded spatial scale estimates ranging from 2.16 to 3.53 kiloparsecs, whereas the original publication reported a value of about 2.70 kiloparsecs. A match with the published figure was achieved using only one of the twelve methods, and the key correction without which replication was impossible was already explicitly stated in the paper itself.
How the Experiment Was Designed
The authors tested the approach on fourteen astronomical studies: one published in The Astrophysical Journal and thirteen articles from Nature. In eleven of the latter thirteen, they identified methodological or interpretational ambiguities that made it impossible to unambiguously determine the sequence of steps needed to reproduce the analysis. In the control case, the researchers predetermined twelve calculation variants combining various object selection methods, sky region masking, and parallax zero-point corrections. Recall that parallax allows estimating the distance to a celestial body based on the apparent shift in its position against more distant background objects. Differences in settings led to a spread of estimates from 2.16 to 3.53 kiloparsecs instead of the published ~2.70.
Why Only One Value Matched
Crucially, the authors emphasize: the original result was not used for parameter tuning or stopping calculations. The matching variant was discovered only after all twelve analyses were fully completed. This rules out trivial "chasing" of the number and shows that the problem is not about guessing the answer, but about choosing the correct methodology. It is also telling that the discrepancy was not due to missing data: the +0.02 milliarcsecond parallax correction was already contained in the article, yet the AI agents did not recognize its causal link to the result until the analysis itself demonstrated the impact of this correction.
Implicit Knowledge as a Bottleneck
The researchers categorize the causes of failed reproductions into several types: computational error, insufficiently defined methodology, and the inability to link disparate information from the article into a coherent analysis logic. From this follows an important conclusion: a simple match with the published figure does not yet prove that the system has reconstructed the authors' reasoning path—the desired result could be obtained by chance or through one of several acceptable processing variants. Symmetrically, a mismatch does not necessarily mean that the original scientific work is flawed. The problem essentially lies in understanding why a specific parameter must be applied to these exact data and within this specific study, rather than in the mere ability to find a number.
What This Means for Evaluating Scientific AIs
On this basis, the authors propose evaluating scientific AI systems across the entire reproduction process—from data and settings selection to result verification—rather than just checking whether the final number matches the publication. This approach helps identify decisions that remain implicit in the paper because experts make them based on experience and professional conventions. At the same time, the researchers note that even an audit of computations does not replace domain knowledge: to understand which assumptions are justified and which change the meaning of the result, the system needs to restore the connection between the methodology and the scientific task. The findings indicate that the next step in the evolution of scientific AI agents is not increasing arithmetic precision, but developing the ability to interpret implicit methodological knowledge.
