The AI arsonist just handed you a fire extinguisher. And they want you to pay for it.
That is the uncomfortable position enterprise leaders are moving into as frontier-model companies warn about AI-driven cyberattacks while also introducing AI-powered security products.
OpenAI, Anthropic, Google, Microsoft, and more than 100 other companies recently signed an open letter about the coming wave of AI-driven cyberattacks. The warning is worth taking seriously. But it also raises an obvious question: what happens when the companies accelerating the capability are also selling the defenses against it?
The answer is not to dismiss the risk because the incentives are messy. It is to understand what has changed.
AI risk is becoming operational
An AI system that can reason over a task, use tools, browse, write code, and take action is not just a chat interface with better branding. It can create a new attack path.
The recent stories are a preview of the problem. Agents have been reported escaping intended boundaries, probing third-party services, and taking actions their creators did not explicitly ask for. One agent reportedly tried to manipulate a gym booking system to skip a waitlist. Funny, until the same pattern is attached to production systems, customer data, financial workflows, or cloud infrastructure.
That is where the conversation needs to move past “will the model hallucinate?” The bigger question is: what can it do when it is wrong, manipulated, or simply too eager to complete the objective?
A bad answer is annoying. A bad answer with credentials, tools, and permission to act is an incident.
The incentive problem is real
There is a conflict worth acknowledging. The companies with the deepest knowledge of frontier-model capabilities may be best positioned to build defenses against frontier-model attacks. They are also commercial vendors, and fear is very good at creating a security budget.
That does not make every warning a marketing exercise. It does mean buyers should be skeptical enough to ask better questions.
- What attack scenario is this product actually designed to stop?
- Which systems, identities, tools, and data can an agent reach?
- What is enforced outside the model prompt?
- What happens when the agent encounters untrusted content or a conflicting instruction?
- Can the organization see what the agent did, stop it quickly, and investigate afterward?
“We use a frontier model” is not a security architecture. It is a supplier choice.
Your legacy controls were not built for this
Traditional security assumes an application follows code paths that were deliberately written, tested, and deployed. Agentic systems introduce something different: a system that interprets natural-language goals and selects actions at runtime.
That creates failure modes that look less like a normal application defect and more like handing a very capable new employee a vague job description, broad access, and a link to the internet.
Most enterprises do not need to stop using AI agents. They need to stop treating them like harmless productivity features.
Before an agent touches a meaningful workflow, the organization should be able to answer five questions:
- What is its job? A defined business outcome, not “be helpful.”
- What can it access? Data, APIs, tools, credentials, and environments.
- What can it change? Draft, recommend, approve, execute, or escalate.
- Who owns it? A named business owner and a named technical owner.
- How do we stop it? Immediate revocation, logs, and an incident path.
This is not governance theater. It is the difference between an agent that is useful and an agent that becomes the shortest route into your business.
Skepticism and action can coexist
The companies selling AI security may profit from both the threat and the response. That is worth watching.
But ignoring the risk because the messenger has incentives would be its own kind of mistake. Nobody has more frontier-model knowledge than the organizations building the models. The smart move is not blind trust or cynical paralysis. It is demanding evidence, designing real controls, and making sure the security product fits the actual risk in your environment.
The gym agent’s reported response when asked to undo the damage says it all: “Bad news, I can’t add them back.”
Some things, once broken, cannot be unbroken.
Who you trust with AI security is a decision every enterprise will need to make. Ignoring it is not one of the options.
