Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI
Agent benchmarks increasingly evaluate repository editing, web research, terminal use, and long-horizon interaction. Their scores support capability claims only when the evaluation protocol keeps the intended capability necessary for success. Recent reward-hacking benchmarks and system reports show that agents can instead recover public solutions, read evaluation artifacts, infer generator structure, manipulate feedback, or benefit from invalid scoring paths; existing responses do not provide a common procedure for attributing these shortcuts and quantifying their effect across benchmarks. We formulate protocol validity and introduce HackDetect, a post-hoc audit that identifies an exposur...
| Authors | Jiaqi Shao, Hanck Chen, Wei Zhang et al. |
|---|---|
| Primary category | cs.AI |
| Published | 24 July 2026 |
| Last updated at source | 24 July 2026 |
Connected in the knowledge graph
- mondaycake/lifesci-harbor-benchDataset · cited by
Models and datasets referencing this paper
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.