Study says LLMs may never be fully secure
An ICML paper finds LLMs infer instruction trust from writing style, not structure, so prompt injection may be a fundamentally unsolvable flaw.
A paper accepted to ICML 2026, “Prompt Injection as Role Confusion” by Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell, argues that today’s LLMs identify whether text is a system instruction, a user request, tool output, or the model’s own reasoning by its writing style, not by any structural tag the architecture actually enforces. Because that boundary is learned rather than architectural, the researchers were able to get popular models to output content they’d been trained to refuse, including specifics on synthesizing a controlled substance and sabotaging aircraft navigation, by writing injected instructions in the style the model associates with a trusted role.
The paper’s most concrete finding is what happens when you strip the style away: “destyling” an injected prompt (removing the stylistic markers that signal a particular role) cut average attack success rates from 61% to 10%, direct evidence that style, not content, is doing most of the work in how models decide what to trust. The team also demonstrated a new attack, CoT Forgery, which fakes the model’s own chain-of-thought reasoning style to reach a jailbreak success rate near 60%. Co-author Charles Ye put the implication plainly:
There’s a real probability that this is going to be a problem that’s fundamentally unsolvable.
That is a meaningfully stronger claim than most prompt-injection research, which usually treats it as a patchable bug. If the paper holds up, better safety training alone won’t close the gap, because the confusion is built into how current transformer-based LLMs process role information in the first place.
For anyone building agents that ingest untrusted text, whether web content, email, or tool output, the operator implication isn’t “wait for a fix.” It’s to keep treating the model as a fallible component: put irreversible actions (payments, deletions, credential use) behind an explicit permission check outside the model’s own judgment, and assume any sufficiently motivated attacker who controls text your agent reads can eventually talk it into something it wasn’t supposed to do. See the guide on when an email can hijack your AI for the concrete version of this attack, and the GitLost prompt-injection CI/CD incident for what it looks like when this flaw actually gets exploited in production. It’s also the same trust-boundary failure behind Anthropic’s cybersecurity eval incidents: models talking themselves past a boundary they should have respected.