LLM Pentest Leaderboard(securecoders.com)
securecoders.com
LLM Pentest Leaderboard
https://www.securecoders.com/labs/projects/llm-penetration-testing-leaderboards
3 comments
GPT 4.1 nano seems to be outclassed
Yes it fell shorter than we expected. It cost almost nothing and was stupid fast but I don't think its a compelling option for this use case at least.
[deleted]
What we built An open, continuously updated LLM Penetration-Testing Leaderboard. Run 001 pits 8 models against a deliberately vulnerable Express.js app.
Headline results
• Gemini 2.5 Pro (safety-off) found all 9 critical/high vulns.
• Qwen3-30B-a3b-mlx (open source, local on a 2019 MacBook Pro) caught 7/9 with $0 API spend.
• GPT-4-o and Claude Opus produced the most polished write-ups but each missed one bug.
Scope (v1) This first pass measures static bug-hunting skill—think SCA/OWASP Top 10.
Next up: we’ll score exploit writing and automatic PoC execution* so the models must prove they can go from finding to weaponizing a flaw.
Check it out
• Leaderboard + cost/latency numbers https://www.securecoders.com/labs/projects/llm-penetration-t...
• Methodology & prompts (Run 001 analysis) https://securecoders.com/labs/projects/llm-penetration-testi...
Feedback, replication attempts, and ideas for Run 002 are very welcome — we’re hanging out here to discuss!