SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models
SciCode is the standard measure of the scientific-coding ability of language models: research-level problems that demand both frontier scientific theory and its implementation as working numerical code. It is a component of the Artificial Analysis Intelligence Index and a standing evaluation in government and national-laboratory suites. Yet its scores have recently plateaued: the strongest 2026 models cluster tightly around 60\% subproblem accuracy, and a successor model ties its predecessor. We trace this stagnation to defects in the benchmark itself. A per-problem, domain-expert audit of all 65 test problems uncovers 263 defects; 192 of them, spread across 91\% of the main problems, cau...
| Authors | Sihan Hu, Lyuhan Huang, Youjin Deng et al. |
|---|---|
| Primary category | cs.SE |
| Published | 5 August 2026 |
| Last updated at source | 5 August 2026 |
Popularity, last 24 days
0 now. Counts as reported by the source, snapshotted daily by Silytics.
Connected in the knowledge graph
- shhu2001/SciCode-VerifiedDataset · cited by
- amd/Mistral-Large-4-1T-A52B-Quark-MXFP4Model · cited by
Models and datasets referencing this paper
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 9 November 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.