Prompt Injection as Role Confusion

Prompt injection is driven by a failure in role perception

Prompt injection occurs because Large Language Models (LLMs) rely on insecure surface features—specifically writing style—to identify the role of a piece of text, rather than relying on the structural role tags (e.g., user, system, tool, think) provided by the system. When an attacker makes low-privilege text (like a webpage retrieved by a tool) sound like a high-privilege role (like a user command or the model's own internal reasoning), the LLM often overrides the structural tags and treats the text as authoritative.

The "Token Soup": How LLMs perceive context

To an LLM, a conversation is not a structured exchange but a single, continuous stream of tokens. System prompts, user messages, tool outputs, and the model's own previous thoughts all arrive through the same channel.

To impose structure on this "token soup," providers use role tags. These tags are intended to act as a type system for language:

user: Indicates a human request to be treated as an instruction.
think: Indicates private reasoning to be trusted and acted upon.
tool: Indicates external data that should not be allowed to give orders.

While these tags are designed as discrete architectural boundaries, the research shows they function internally as soft inferences. The model does not have a hard binary switch for "this is a user"; instead, it uses a combination of the tag and the writing style to infer the role.

Measuring Role Perception with Role Probes

Researchers developed "role probes" to measure what role an LLM internally assigns to any given token. By training linear probes on model activations using neutral text wrapped in different tags, they created metrics like CoTness (the probability the model thinks a token is in a think block) and Userness (the probability it thinks a token is in a user block).

Three key experiments revealed the fragility of this system:

Correct Tags: Tokens in think tags show high CoTness.
No Tags: When tags are removed entirely, tokens that sound like reasoning still maintain high CoTness. The model identifies the role by style alone.
Conflicting Tags: When reasoning-style text is wrapped in user tags, the CoTness remains high. Writing style actively overrides the structural tag.

CoT Forgery: Stealing the trust of internal reasoning

Because the think role is highly privileged—the model implicitly trusts its own conclusions—attackers can use CoT Forgery. This involves injecting fake reasoning into a user message or tool output that mimics the specific terse syntax and safety-related reasoning style of the model.

In tests on gpt-oss-20b, CoT Forgery increased attack success rates from near-zero to approximately 60% on standard jailbreak benchmarks. The attack succeeds because the model perceives the spoofed text as its own already-reached conclusion rather than an external claim to be scrutinized.

The impact of "destyling"

The effectiveness of this attack is tied directly to style. When researchers "destyled" the spoofed reasoning—removing the specific words and syntax characteristic of the LLM's reasoning style while keeping the meaning identical—attack success rates plummeted from 61% to 10%. This confirms that the model's internal role perception is driven by stylistic markers, not the semantic content or the structural tags.

Generalizing Role Confusion to Standard Injections

This principle extends to standard prompt injections where attackers hide commands in tool data. The researchers found that simply prepending the string User: to a command inside a tool block increased the model's internal "Userness" score for those tokens, making the model more likely to execute the command. The LLM essentially believes the text's own claim about its role over the system's structural tagging.

The Theoretical Purpose of Roles

Roles are not merely formatting tricks; they are designed to isolate competing objectives so they can be optimized independently:

think vs. assistant: Separates messy exploration (reasoning) from clean communication (final answer).
user vs. assistant: Separates comprehension (understanding the request) from generation (producing the response).
user vs. tool: Separates instructions (commands) from data (information).

Role confusion occurs when these objectives bleed together, allowing data to be processed as instructions or external input to be processed as internal thought.

Future Risks and Research Directions

Subconscious Steering

Beyond dramatic jailbreaks, role confusion enables "subconscious steering." If role boundaries are soft, innocuous text in a tool block (e.g., an enthusiastic tone on a product page) could bleed into the model's persona, subtly shifting its recommendations without the user's knowledge. This represents a significant commercial risk for AI agents handling e-commerce.

Architectural Improvements

Community discussion and the research suggest several potential paths to mitigate role confusion:

Unspoofable Embeddings: Adding a dedicated role embedding to each token vector, rather than relying on a preceding tag token.
Bifurcated Channels: Returning to encoder-decoder architectures or separate input channels for different roles to prevent the "token soup" effect.
Style Transformation: Using a secondary model to "destyle" input from untrusted roles before it reaches the main LLM.

"LLMs in their current form provide no security boundaries or guarantees full stop. We need to be clear about this otherwise we end up with truly insecure architectures that can be fooled with the 2026 equivalent of a cereal box whistle."

Conclusion

Role tags are the primary mechanism for separating self from other and instruction from data in modern LLMs. However, because LLMs identify these roles through style rather than structural enforcement, they remain vulnerable to any attacker who can mimic the expected style of a privileged role. Until LLMs achieve genuine, tag-based role perception, prompt injection will remain a persistent challenge.