Attention Bottlenecks for Multimodal Fusion

Abstract

Humans perceive the world by concurrently processing and fusing high-dimensional inputs from multiple modalities such as vision and audio. Machine perception models, in stark contrast, are typically modality-specific and optimised for unimodal benchmarks, and hence late-stage fusion of final representations or predictions from each modality (`late-fusion') is still a dominant paradigm for multimodal video classification. Instead, we introduce a novel transformer based architecture that uses `fusion bottlenecks' for modality fusion at multiple layers. Compared to traditional pairwise self-attention, our model forces information between different modalities to pass through a small number of bottleneck latents, requiring the model to collate and condense the most relevant information in each modality and only share what is necessary. We find that such a strategy improves fusion performance, at the same time reducing computational cost. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens and VGGSound. All code and models will be released.

Published at NeurIPS 2021. Note this version updates numbers due to a bug in the AudioSet mAP calculation in Table 1 (last row)

References in corpus (11)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• The Kinetics Human Action Video Dataset
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
• Perceiver: General Perception with Iterative Attention
• Learning to Generate Diverse Dance Motions with Transformer
• Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification
• Cross-Modal Self-Attention Network for Referring Image Segmentation
• AST: Audio Spectrogram Transformer
• Parameter Efficient Multimodal Transformers for Video Representation Learning
• Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds

Cited by in corpus (12)

• Neural Entity Linking: A Survey of Models Based on Deep Learning
• Materials science in the era of large language models: a perspective
• Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers
• Integrated Brain Connectivity Analysis with fMRI, DTI, and sMRI Powered by Interpretable Graph Neural Networks
• Fusion of Satellite Images and Weather Data with Transformer Networks for Downy Mildew Disease Detection
• Exploring Attention Mechanisms for Multimodal Emotion Recognition in an Emergency Call Center Corpus
• Multi-modal Extreme Classification
• MM-ALT: A Multimodal Automatic Lyric Transcription System
• Scalable and Accurate Self-supervised Multimodal Representation Learning without Aligned Video and Text Data
• SCENIC: A JAX Library for Computer Vision Research and Beyond
• Pelta: Shielding Transformers to Mitigate Evasion Attacks in Federated Learning
• How Intermodal Interaction Affects the Performance of Deep Multimodal Fusion for Mixed-Type Time Series