Omni-MMSI: Toward Identity-attributed Social Interaction Understanding
We introduce Omni-MMSI, a new task that requires comprehensive social interaction understanding from raw audio, vision, and speech input. The task involves perceiving identity-attributed social cues (e.g., who is speaking what) and reasoning about the social interaction (e.g., whom the speaker refers to). This task is essential for developing AI assistants that can perceive and respond to human interactions. Unlike prior studies that operate on oracle-preprocessed social cues, Omni-MMSI reflects realistic scenarios where AI assistants must perceive and reason from raw data. However, existing pipelines and multi-modal LLMs perform poorly on Omni-MMSI because they lack reliable identity att...
| Authors | Xinpeng Li, Bolin Lai, Hardy Chen et al. |
|---|---|
| Primary category | cs.CV |
| Published | 31 March 2026 |
| Last updated at source | 31 March 2026 |
Connected in the knowledge graph
- Xinpeng-Li/Omni_MMSIDataset · cited by
Models and datasets referencing this paper
- Xinpeng-Li/Omni_MMSIDataset
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.