On Randomness in Agentic Evals
Agentic systems are evaluated on benchmarks where agents interact with environments to solve tasks. Most papers report a pass@1 score computed from a single run per task, assuming this gives a reliable performance estimate. We test this assumption by collecting 60,000 agentic trajectories on SWE-Bench-Verified, spanning three models and two scaffolds. We find substantial variance: single-run pass@1 estimates vary by 2.2 to 6.0 percentage points depending on which run is selected, with standard deviations exceeding 1.5 percentage points even at temperature 0. This variance has critical implications: reported improvements of 2--3 percentage points may reflect evaluation noise rather than ge...
| Authors | Bjarni Haukur Bjarnason, André Silva, Martin Monperrus |
|---|---|
| Primary category | cs.LG |
| Published | 6 February 2026 |
| Last updated at source | 25 March 2026 |
Popularity, last 2 days
0 now. Counts as reported by the source, snapshotted daily by Silytics.
Connected in the knowledge graph
- ASSERT-KTH/agentic-evals-artifactsDataset · cited by
Models and datasets referencing this paper
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 19 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.