Intervention, Not Shared Latents: Blocking Visual Shortcuts in Audio-Video Generation
Joint audio--video (AV) generators are trained on data in which \emph{what an event looks like} and \emph{what it sounds like} are spuriously correlated. We present a \emph{controlled causal study} of the resulting failure mode. In an AV structural causal model where the audio is, by construction, independent of the video's nuisance appearance, models that let audio read video directly---through cross-attention or a shared latent---learn a \emph{visual shortcut}: they predict sound from appearance rather than the causal event and, when the appearance--event correlation is broken at test time, synthesize the wrong event's sound. Crucially, the popular remedy of routing both modalities thro...
| Authors | Jian Xu, Delu Zeng, John Paisley |
|---|---|
| Primary category | cs.LG |
| Published | 14 August 2026 |
| Last updated at source | 23 September 2026 |
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.