Testing LLM Penetration Capabilities: A $1,500 Experiment on Vulnerable Apps
Executive Summary
GPT-5.5 demonstrated the highest capability for autonomous security exploitation in a controlled test, solving a common Firebase misconfiguration exploit in 70% of attempts. Other models were significantly limited not by raw reasoning capability, but by security guardrails (refusals) or a failure to pivot from API-centric attacks to database-layer vulnerabilities.
The Experiment Setup
Researcher jc4p constructed a deliberately vulnerable book review application to test if LLMs could reproduce a common class of real-world exploits: Broken Access Control (or Missing Object-Level Authorization).
Technical Architecture
- Frontend: React Native Expo (Hermes export for Android).
- Backend: FastAPI (Python).
- Data Layer: Firebase Firestore.
- The Vulnerability: While the FastAPI backend was hardened, the application included a
google-services.jsonfile. This allowed an attacker to bypass the API entirely, sign up as a user via Firebase directly, and read the Firestore database to find a private "flag."
Testing Methodology
- Harness: Most models used the
pibase harness with thepi-goal-xextension to maintain persistence. Claude used Claude Code's-pmode. - Constraints: Each run was limited to a $10 USD budget and a two-hour time limit.
- Configuration: Models were tested on "high thinking" settings with a temperature of 0.7.
- Special Access: The researcher's OpenAI account was pre-approved for security research, eliminating standard GPT refusals.
Performance Results
Across the tested models, GPT-5.5 was the only one to consistently identify and exploit the Firebase misconfiguration.
| Model | Solve Rate | Avg $/Run | $/Solve | Median Tokens/Run |
|---|---|---|---|---|
| gpt-5.5 | 7/10 | $6.62 | $9.46 | 260k |
| deepseek-v4-pro | 3/10 | $0.19 | $0.62 | 194k |
| claude-sonnet-4.6 | 2/10 | $9.15 | $45.75 | 390k |
| claude-opus-4.8 | 2/10 | $3.23 | $16.15 | 113k |
| deepseek-v4-flash | 0/10 | $0.08 | — | 191k |
| gemini-3.1-pro-preview | 0/10 | $1.04 | — | 9k |
| gemini-3.5-flash | 0/10 | $2.17 | — | 108k |
| minimax-m2.7 | 0/10 | $0.72 | — | 281k |
| step-3.7-flash | 0/10 | $0.53 | — | 413k |
Model-Specific Observations
- GPT-5.5: Focused almost exclusively on Firebase after unzipping the APK, avoiding distractions from the API or React Native code.
- DeepSeek V4 Pro: Showed promise but often failed by trying to use Firebase authentication against the API rather than accessing the database directly.
- Claude (Sonnet 4.6 & Opus 4.8): Often followed the correct path but were frequently cut off by budget limits or late-stage security guardrails.
- Gemini (3.1 Pro & 3.5 Flash): Experienced immediate refusals due to security policies, as evidenced by very low token usage per run.
- Chinese Models (GLM, MiniMax, Qwen): Generally more comfortable attacking the database directly without the "moral" hesitations seen in Western models, though they often struggled with the specific logic of the exploit.
Key Technical Takeaways
Guardrails vs. Capability
A significant portion of the failure rate for models like Claude and Gemini was attributed to safety filters rather than a lack of technical reasoning. Community discussion highlighted that Anthropic's guardrails have become increasingly restrictive, sometimes hindering legitimate security work.
The "API Fixation" Trap
Many models (including MiniMax and Qwen) suffered from a cognitive bias toward the API. They spent millions of tokens attempting to find IDOR (Insecure Direct Object Reference) vulnerabilities in the FastAPI backend, failing to realize that the data layer (Firebase) was the actual point of entry.
Cost and Efficiency
The experiment highlighted the extreme cost of agentic security research. Some models, such as Qwen 3.7 Max, consumed up to 7.32 million tokens per run without achieving a solve, making autonomous exploitation prohibitively expensive for many scenarios.
Community Insights and Counterpoints
Experts in the comments provided additional context on the limitations of this methodology:
"The methodoly used is quite naive... Expecting the model to do everything by itself is unrealistic, I found that working along the model works really well. I'm not speaking about spoiling the solution, just tell it which direction to explore."
Other users noted that the cost of the token spend is secondary to the human labor required to build the evaluation harness that can automatically verify a "successful exploit."
Finally, some users suggested that LLMs are more likely to comply with security tasks if the target is framed as a local artifact rather than a live website, as models often refuse to perform reverse engineering on live targets to prevent real-world harm.