I benchmarked LLM agents on fixing real-world security vulnerabilities

Show HN: I benchmarked LLM agents on fixing real-world security vulnerabilities

I benchmarked LLM agents on fixing real-world security vulnerabilities is a business in show hn launch.

Profile

I built a benchmark with 20 real CVEs across 18 Python projects (Pillow, GitPython, yt-dlp, urllib3, etc). I've run it over 5 LLM agents (3 OpenAI, 2 poolside) and 3 different prompts (full advisory, locate, diagnose) with a total of 300 runs. The agents are tasked to fix security vulnerabilities in a sandboxed environment and they are scored against a hidden security tests from the maintainer's own fix.Best solve rate was 50%. On the other 50%, some fixes are sometimes coherent and pass all regression tests, but vulnerability still present.The main differentiator I found between models is cost: gpt-5.5 at 12× more expensive than gpt-5.4-mini while producing statistically similar results. Wi

Within the show hn launch space, the company focuses on its core offering rather than a broad product line.

Its current recorded stage is Show HN launch.

Website: giovannigatti.github.io/cve-bench