Beyond Ideal Instruction: A Comprehensive Framework for Evaluating LLMs in Realistic Interactions
Despite great advances in tool-use capabilities of large language models (LLMs), existing evaluation benchmarks struggle to fully align with real-world scenarios. Such benchmarks mostly rely on simulated idealized user assumptions and lacks experience-oriented evaluation. These limitations fail to account for the ambiguity, uncooperative behaviors, and shifting intentions characteristic of real-world users. To fill this gap, we propose RUT-Bench, a dedicated benchmark designed to assess LLMs under diverse Real-world User Tool calling scenarios. RUT-Bench supports high-fidelity simulations covering both ideal rational patterns and heterogeneous non-ideal behaviors across single-turn and mu...
| Authors | Xuan Yang, Hao Xu, Tingfeng Hui et al. |
|---|---|
| Primary category | cs.CL |
| Published | 2 June 2026 |
| Last updated at source | 3 June 2026 |
Models and datasets referencing this paper
- Miaow-Lab/RUT-BenchDataset
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.