Ignoring the Rules: ChatGPT-User Bypasses robots.txt Ban in 54% of Cases

In the first half of 2026, a worrying trend has been recorded in the sphere of interaction between artificial intelligence and web content. According to data from the TollBit platform, the ChatGPT-User agent used by OpenAI accessed pages that website owners had explicitly forbidden via the robots.txt file in 54% of recorded cases. This figure is the highest among all analyzed bots, raising questions about the effectiveness of traditional methods for managing access to web resources.

European Market: Leading in Violations

Statistics show that the problem is most acute in the European segment of the internet. In the sample of European websites, the violation rate for ChatGPT-User reached a peak. For comparison, other popular bots demonstrate lower but still significant figures: Bytespider bypassed bans in 48% of cases, and PerplexityBot in 42%. Overall, across a sample of European and North American websites, about 15% of all AI bot requests were directed at pages with an explicit ban in robots.txt. This indicates a systemic nature of the problem that goes beyond a single specific service.

Technical Specifics and OpenAI's Position

It is important to understand the context of the ChatGPT-User bot's operation. According to official OpenAI documentation, this agent is activated when a user asks a question requiring the retrieval of current content from a specific web page. OpenAI emphasizes that the agent is not intended to automatically bypass bans in the background. The company points out that the request is initiated directly by the user, and therefore robots.txt rules may not apply to it. This fundamentally distinguishes ChatGPT-User from OAI-SearchBot, which determines the possibility of using content in search results, and GPTBot, which is designed for collecting data to train base models.

Contradictory Data

There is a discrepancy between technical implementation and webmasters' expectations. On the one hand, publishers rely on robots.txt as the primary tool for controlling visibility, expecting bots to respect it. On the other hand, platforms like OpenAI interpret the rules differently, considering user-initiated requests to take precedence over robots.txt directives. Furthermore, TollBit statistics record the fact of access to forbidden URLs, but this alone cannot establish the true cause of each request or rule out the possibility of user-agent spoofing by malicious actors, creating uncertainty in the interpretation of the figures.

The End of the robots.txt Era as Protection

Data from 2026 confirms that the robots.txt file has ceased to be a technical protection for a page against content retrieval. It is merely a tool for conveying instructions to automated agents, but it does not prevent the server from serving the file upon request. If a resource truly must not be provided to an external agent, it is necessary to implement real technical access control mechanisms: authentication, authorization, network rules, or server-level blocking. For teams managing web resources, this means the need to review security strategies and separate the concepts of 'visibility for search engines' and 'actual access to data'.