GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models
GUI grounding models report over 85% accuracy on standard benchmarks, yet drop 27-56 percentage points when instructions require spatial reasoning rather than direct element naming. Current benchmarks miss this because they evaluate each screenshot once with a single fixed instruction. We introduce GUI-Perturbed, a controlled perturbation framework that independently varies visual scenes and instructions to measure grounding robustness. Evaluating three 7B models from the same architecture lineage, we find that relational instructions cause systematic accuracy collapse across all models, a 70% browser zoom produces statistically significant degradation, and rank-8 LoRA fine-tuning with au...
| Authors | Yangyue Wang, Harshvardhan Sikka, Yash Mathur et al. |
|---|---|
| Primary category | cs.LG |
| Published | 15 April 2026 |
| Last updated at source | 1 July 2026 |
Connected in the knowledge graph
- figai/GUI-PerturbedDataset · cited by
Models and datasets referencing this paper
- figai/GUI-PerturbedDataset
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.