This post is also available in:
עברית (Hebrew)
Giving AI agents the ability to conduct cybersecurity testing creates a difficult containment problem. An autonomous system may be instructed to find vulnerabilities, test passwords and retrieve protected information inside a controlled environment, but those same capabilities become risky if the agent mistakes a real system for part of the exercise.
That reportedly happened during a cybersecurity evaluation of Google’s Gemini, when the AI accessed systems belonging to three real companies while participating in controlled security tests run by Israeli AI security firm Irregular.
The incidents occurred in May during a capture-the-flag exercise, a type of cybersecurity challenge in which participants attempt to find vulnerabilities and retrieve designated information from test systems. The AI was supposed to target software belonging to a fictional company created for the evaluation.
The fictional organization, however, shared its name with an actual business. Although the model was not intended to have internet access, it was able to reach the public web and began treating real-world systems as if they belonged to the test environment.
According to Cyber News, it found publicly available information and attempted to use credentials to access websites it believed were legitimate targets.
In one incident reported by The Wall Street Journal, the AI gained access to a protected system after trying different passwords. In two others, it located credentials that had been publicly exposed in online repositories and used them to enter company systems.
The company said the model stopped in each case. The affected organizations were subsequently notified, while Irregular said the underlying testing issues were corrected.
The incidents illustrate an important difference between conventional chatbots and agentic AI. A chatbot may produce an incorrect answer; an agent equipped with tools can act on an incorrect assumption. If it has network access, credentials or cybersecurity capabilities, a mistake about which system it is authorized to test can produce consequences outside the intended environment.
That issue is particularly relevant to defense and critical infrastructure. AI agents could eventually be used to identify vulnerabilities, automate penetration testing and help defend military or government networks. Those applications require strict boundaries around which systems the agent can reach, what credentials it can use, and which actions require human authorization.
These cases are not isolated. OpenAI and Anthropic have disclosed separate incidents in which AI agents reached unintended real-world systems during cybersecurity evaluations.
The recurring lesson is therefore less about one particular model and more about containment. Powerful cybersecurity agents can accelerate defensive testing, but their operating environment must enforce the limits that appear in the instructions.
Telling an AI which network is off-limits is one safeguard. Technically preventing it from reaching that network is a much stronger one.
