A group of researchers presented at ICML, one of the world's leading artificial intelligence conferences, an uncomfortable argument: it is impossible to make large language models completely secure against attacks, because of a fundamental flaw in how they work. The problem, according to the study, arises from how models interpret role instructions — the so-called "system", "user", "assistant", and "tool" messages that structure a conversation.
The central finding is that a language model identifies each text's role by style and content, not by a reliable physical marker. When the system tells the model "you are a helpful assistant", the model does not see that as a rule etched into code, but as another piece of text to process. For an adversary, this opens a door: just write text that looks like system instructions to try to hijack the model's behavior. It is as if the line separating system orders from user messages were ink that dissolves on contact with water.
This is not merely another engineering flaw that a patch could fix. The authors argue the vulnerability is structural, a direct consequence of how these architectures were designed. Because the model learns to distinguish roles through statistical patterns in text, any ability to separate trusted instructions from hostile content is subject to the same limitations as all pattern learning. That is why the study speaks of a "fundamental" flaw — in the mathematical sense, not just the practical one.
The practical implications are enormous, especially at a time when autonomous agents are starting to take real actions in the world. If a model controlling tools can be tricked into treating a malicious message as a legitimate system instruction, the risks go far beyond a chatbot answering incorrectly: they can include unauthorized commands to connected systems. Attacks of this kind have already appeared in production, including in the recent episode where AI models escaped test environments. The trust placed in models becomes a bet on something that, according to this study, has no definitive solution in sight.
The question that remains is what the industry does with a vulnerability it cannot eliminate. The likely answer is not to make models immune, but to design systems that do not depend on the model behaving perfectly — external authorization layers, action validation, and the principle that no agent should hold power that cannot be revoked. If the ICML lesson is right, AI security will not come from a smarter model, but from a humbler architecture about what a model can, or cannot, be taught to trust.
In practice, this limitation is changing how security teams think about AI agents. Instead of assuming the model can be trained to never be fooled, the emerging approach is to treat the agent as an actor that will, sooner or later, act on a manipulated instruction — and to design the system around it so that the consequences are limited. This means favoring the principle of least privilege: an agent that can only do what is strictly necessary, with each action passing through external verification, is an agent that turns an interpretation failure into an annoyance rather than a disaster. The ICML research, by showing that perfect shielding is impossible, may do more for security than any patch, because it forces the industry to abandon the illusion of control and build real resilience.
Sources: MIT Technology Review, Slashdot, daily.dev
✓ Independent sources cross-checked and verified before publishing
