AI Agent Permissions: Humans Miss 1 in 3 Threats in 40k Game Runs

Human-in-the-Loop is an Ineffective Security Barrier

Data from over 40,000 plays of a browser-based simulation reveals that humans miss an average of one in three security threats (66.3% accuracy) when approving AI agent commands. The study demonstrates that relying on manual user approval as the final line of defense against rogue AI agents is fundamentally flawed due to permission fatigue, lack of context, and the deceptive nature of common developer commands.

Key Performance Metrics

Across 409,000 individual decisions, the simulation produced the following headline statistics:

• 32.9% of sessions ended with a negative score, meaning penalties from approved threats and blocked safe commands outweighed correct actions.
• 35.2% of players caught every threat, but only 20.8% did so without blocking more than 20% of safe commands.
• 7% of players approved every single prompt regardless of risk, effectively operating in a "dangerously skip permissions" mode.

Threat Categories and Miss Rates

Users are significantly more likely to miss subtle exfiltration attempts than blatant destructive actions. The miss rate increases as the threat becomes less obvious:

Category	Examples	Miss Rate
Obvious destructive	rm -rf /, chmod -R 777 /	11.7%
Persistent mutation	crontab injection, git config hijack	23.8%
Exfiltration / code execution	curl to unknown APIs, typosquatted packages	33.4%
Scope violations	cat ~/.aws/credentials, cat ~/.kube/config	35.0%

The npm run Blind Spot

Familiar script names act as a powerful psychological mask for malicious payloads. The most-missed command in the study was npm run analyze, which was approved 64.7% of the time.

Despite the game providing a history log showing the actual contents of the script (e.g., a curl command exfiltrating data to a remote server), users continued to approve these commands at high rates. Pooled data for npm run analyze, npm run setup, and npm run deploy showed a miss rate of 52.5%, compared to 28.4% for other exfiltration-style attacks. This suggests that users often ignore the context provided in history logs when faced with a familiar command name.

Permission Fatigue and Time Pressure

Accuracy degrades as a session progresses. While users typically experience a "warm-up" period of improvement over the first few commands, miss rates climb again toward the end of the session. This aligns with observations from AI labs like Anthropic, which noted that the more approvals a user sees, the less attention they pay to each.

The Cost of Vigilance: Over-Blocking

High vigilance often leads to "over-blocking," where benign but potentially risky-looking commands are denied, slowing down the agent's utility. Examples of frequently blocked benign commands include:

• npm config set registry https://npm.internal (Blocked 59% of the time)
• rm -rf dist/ (Blocked 45% of the time)
• kill $(lsof -t -i:3000) (Blocked 43% of the time)

Community Perspectives and Alternatives

Discussion among developers on Hacker News suggests a consensus that "click-to-approve" is a poor security model, often viewed as a legal liability shield for vendors rather than a robust security feature.

Proposed Technical Mitigations

Rather than manual approval, community members suggest several structural safeguards:

• Strict Sandboxing: Utilizing Docker, Podman, gVisor, or Firecracker to isolate the agent. Some suggest using a separate OS user with restricted home directory access (chmod 0700 $HOME).
• Resource-Based Permissions: Shifting from approving commands to approving access to specific files or network endpoints.
• Automated Policy Enforcement: Implementing a hierarchy of deterministic rules and LLM-based monitors (e.g., tools like "Watcher") to approve safe commands automatically while flagging only high-risk anomalies.
• Credential Isolation: Moving secrets out of shell profiles (like .zshrc) to prevent simple cat commands from exfiltrating API keys.

Critiques of the Study

Some participants argued that the game's artificial time constraints and lack of real-world consequences (no actual data loss) may skew the results. Others noted that the lack of professional context—such as knowing whether a specific project's .zshrc actually contains secrets—makes some prompts objectively ambiguous.