Same Patient, Different Order: Action-Level Reliability of Clinical LLM Agents Under Repeated Runs
A clinical agent benchmark can report the same verdict on identical inputs while the agent files a materially different order on each run. Such agents order tests, request medications and place referrals, yet benchmarks typically score one run per task and rarely ask whether identical inputs produce identical actions; MedAgentBench, the benchmark we use, scores a single attempt and says so. To measure this gap we introduce "same-input rerun", which replays a task with every input held fixed and compares the orders rather than the score, with six reliability metrics, and apply it to 1000 MedAgentBench runs across 50 tasks from its five write-capable families, two open-weight models below t...
| Authors | Rohith Reddy Bellibatlu, Manpreet Singh, Zhoutian Han et al. |
|---|---|
| Primary category | cs.CL |
| Published | 11 September 2026 |
| Last updated at source | 11 September 2026 |
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.