Google Research: Neural Networks Store 95–98% of Facts but Cannot Retrieve Up to a Third of Them

Researchers at Google Research published a paper showing that state-of-the-art large language models (LLMs) encode 95 to 98% of the tested facts, yet when asked directly they are unable to retrieve 26–34% of them. The authors' key conclusion is that this is not an error in the knowledge itself, but a failure in the mechanism for accessing already stored information. In other words, the model 'knows' the fact but cannot 'pull it out' without additional cues. This fundamentally changes the approach to diagnosing fact-generation errors: the problem shifts from the plane of training and data volume to the plane of retrieval architecture.

The 'Knowledge Profiling' Method

To systematize the problem, the authors proposed a method called knowledge profiling. Instead of evaluating the accuracy of each question in isolation, the system classifies each fact into five categories: not encoded; encoded but not retrieved; retrieved directly; retrieved with reasoning; inferred without encoding. This taxonomy allows real knowledge gaps to be separated from technical access limitations. To build the sample, the researchers used 2,150 facts from 'Wikipedia,' each accompanied by 10 formulated questions, yielding more than 21,000 'fact–question' pairs for analysis.

The 'Empty Shelf' and 'Lost Key' Metaphors

The central metaphor of the paper is simple and intuitive. An 'empty shelf' means that the fact never made it into the model at all — it is not present in its parameters. A 'lost key' means that the fact is present, but the model cannot find it without additional cues. The difference between these two states is fundamental from an engineering standpoint: in the first case, a larger model or a broader coverage of training data is required; in the second, it is enough to improve the methods for retrieving existing knowledge without touching the model itself. This is precisely why accurate diagnosis of the error class is critically important for optimizing the costs of system development.

The Word-Order Paradox

The most unexpected result concerns word order. When a question reverses the familiar 'subject–object' direction, retrieval quality drops sharply. For example, a model easily answers a question about who wrote a particular book, but may fail to handle the rephrasing 'which book did this author write.' The authors emphasize that this is not a knowledge gap but a limitation of the access mechanism: the model stores the fact in a particular orientation and struggles to retrieve it in the reverse direction. At the same time, the Google Research paper does not claim that word order is the sole cause of failures, but rather provides a structured method for measuring this specific class of errors.

Practical Takeaways for Developers

For developers building applications based on language models, the paper's findings are practical in nature. Factual systems must be tested on rephrased questions, with different word orders, and in a recognition format, rather than relying on a single canonical phrasing template. The knowledge profiling method allows the share of 'lost keys' to be quantified and targeted work to be done on improving retrieval. Published against the backdrop of the active deployment of LLMs in corporate and consumer products, the work sets a new standard for verifying fact generation: accuracy on 'convenient' questions is no longer a sufficient criterion for model reliability.