On the Limits of Scale: Inverse Scaling in Vision-Language Models
Scaling laws suggest that increasing model size, training data, and compute generally yields monotonic performance improvements. While this often holds at the aggregate level, we present systematic evidence that vision-language models (VLMs) exhibit inverse scaling at the subtask level: specific question categories on which larger models perform worse than smaller counterparts within the same model family. We evaluate 19 models from four distinct model families across eight multimodal benchmarks (nine evaluation splits), answering 29,386 questions per model, 558,334 model-question evaluations in total. Using per-question pairwise analysis across all model pairs in each family, we identify...
| Authors | Rohit Saxena, Alessandro Suglia, Marek Rei et al. |
|---|---|
| Primary category | cs.CV |
| Published | 19 October 2026 |
| Last updated at source | 19 October 2026 |
Recent changes
- 21 Oct 2026 Description: Scaling laws suggest that increasing model size, training data, and compute generally yields monotonic performance improvements. While this often holds at the aggregate level, we present systematic evidence that vision-language models (VLMs → Scaling laws suggest that increasing model size, training data, and compute generally yields monotonic performance improvements. While this often holds at the aggregate level, we present systematic evidence that vision-language models (VLMs
- 21 Oct 2026 Topics added cs.CV, cs.AI removed hf-daily-papers, vision-language models, inverse scaling, multimodal benchmarks, per-question pairwise analysis, multi-image spatial reasoning, visual perception, stem knowledge, diagram understanding
Changes detected by Silytics between refreshes of the source.
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 21 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.