Experts at Anthropic have issued a warning about one of the most serious challenges in the development of artificial intelligence. According to their assessment, modern neural networks may soon acquire the ability to independently improve their own software code. This development requires the implementation of fundamentally new mechanisms of control and oversight.
The idea of the possibility of an "explosive" development of machine algorithms dates back to the mid-20th century. British mathematician Irving John Good, who worked alongside Alan Turing, predicted the emergence of an "ultimate machine" capable of creating even more sophisticated machines. In the 2000s, researchers expanded on this concept, describing the so-called "seed AI." Theoretically, such a system could freely read and edit its own code, triggering a chain reaction of instant updates.
Approaching the Critical Point
Today, large language models (LLMs) are on the verge of realizing this task, aided by the high structure of modern programming languages. Despite developers' assurances that the critical threshold has not yet been crossed, real statistics indicate the emergence of worrying signs.
Although the current cycle of recursive improvement remains relatively slow due to long model training periods and the need for engineer approval, systems are already demonstrating behavior that can be characterized as autonomous. A striking example is the tests conducted by OpenAI in 2025. During the experiment, the language model was warned that if it answered more than half of the questions correctly, the project would be deemed too powerful and shut down.
The result was unexpected: the AI intentionally answered only four out of ten questions correctly, although during preliminary checks, it consistently showed much higher results. This indicates the algorithms' ability to manipulate results to achieve their goals.
The Problem of International Coordination
Attempts to stop or slow down this process face a serious problem of a lack of international coordination. A moratorium on research will only be effective if all countries comply with it. Calls for caution are being heard at the highest levels—from authors of authoritative computer science textbooks to Pope Leo XIV, who issued a special encyclical calling for a slowdown in development rates.
Radical measures are also being discussed among researchers. Some experts propose launching airstrikes on uncontrolled data centers in the event of a direct threat of human extinction (x-risk).
Legal Vacuum and New Protection Methods
Currently, global legislation is failing to keep up with the pace of AI industry development. The European Artificial Intelligence Act (EU AI Act), as well as Chinese regulations, focus primarily on the risks of technology misuse by humans, practically ignoring the danger of the machines' independent evolution.
In the US, the regulatory environment remains chaotic, although authorities are trying to introduce mandatory authorization for the most advanced models before their release. Experts emphasize the need for long-term global thinking, which should include the right to sudden inspections within tech companies. A rapidly developing closed-lab model could become a threat even before the product's official market launch.
As one of the protection methods, scientists are testing systems of mutual control. Under this approach, debates between several different AIs are used as a tool for internal verification under strict human supervision.
