Covering Human Action Space for Computer Use: Data Synthesis and Benchmark
Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude. Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust. Our analysis of failure cases from advanced models suggests a long-tail pattern in GUI operations, where a relatively small fraction of complex and diverse interactions accounts for a disproportionate share of task failures. We hypothesize that this issue largely stems from the scarcity of data for complex interactions. To address this problem, we propose a new benchmark CUActSpot for evaluating models' capabilities on complex interactions across five modalities: GUI, text, table, canvas, and natural imag...
| Authors | Miaosen Zhang, Xiaohan Zhao, Zhihong Tan et al. |
|---|---|
| Primary category | cs.CV |
| Published | 12 May 2026 |
| Last updated at source | 12 May 2026 |
Connected in the knowledge graph
- microsoft/Phi-Ground-AnyModel · cited by
Models and datasets referencing this paper
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.