TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
Multimodal large language models (MLLMs) should exhibit cross-modal consistency, generating consistent responses given semantically equivalent inputs across modalities. However, we observe a systematic discrepancy in model predictions under such cross-modal variations. Specifically, we define the modality gap as the difference in model performance under semantically equivalent textual and multimodal inputs. We introduce TokenSwap, a method that constructs such inputs by replacing textual concepts with semantically aligned images, resulting in sequences where visual tokens are interleaved with text tokens. Based on TokenSwap, we transform existing text-based benchmarks (e.g., MMLU) into im...
| Authors | Andong Hua, Colton Bishop, Igor Mordatch et al. |
|---|---|
| Primary category | cs.CL |
| Published | 20 May 2026 |
| Last updated at source | 8 October 2026 |
Models and datasets referencing this paper
- dongx1997/TokenSwap-BenchDataset
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.