OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments
Current benchmarks for graphical user interface (GUI) agents predominantly rely on static screenshots. However, real-world smartphone interaction routinely requires agents to process transient audio cues and temporal video dynamics that are tightly coupled with the moment of action. To bridge this gap, we introduce OmniGUI, the first step-level benchmark designed to evaluate GUI agents in omni-modal smartphone environments. OmniGUI provides continuous, interleaved multimodal inputs comprising static images, synchronous audio, and video clips at every action step. The dataset encompasses 709 expert-demonstrated episodes (2,579 action steps) across 29 applications, systematically annotated ...
| Authors | Felix Henry, Lihan Lin, Jiangyou Zhu et al. |
|---|---|
| Primary category | cs.HC |
| Published | 3 April 2026 |
| Last updated at source | 28 September 2026 |
Connected in the knowledge graph
- OmniGUI/OmniGUIDataset · cited by
Models and datasets referencing this paper
- OmniGUI/OmniGUIDataset
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.