SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?
Finding a real vulnerability in complicated systems is a challenging, long-horizon task that demands reasoning across an entire codebase to produce a working proof-of-concept (PoC). However, such critical security problems remain understudied. We present SEC-bench Pro, a benchmark that measures how well frontier models hunt real vulnerabilities by reproducing working PoC inputs from disclosed reports, where each task pairs a concrete bug with the instructions for triggering it. We also demonstrate the limitations of existing rule-based judges for grading generated PoCs, and propose a novel LLM-based judge for more precise grading. We instantiate SEC-bench Pro with 344 validated vulnerabil...
| Authors | Hwiwon Lee, Jiawei Liu, Dongjun Kim et al. |
|---|---|
| Primary category | cs.CR |
| Published | 26 May 2026 |
| Last updated at source | 20 July 2026 |
Models and datasets referencing this paper
- SEC-bench/SEC-bench-ProDataset
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.