MSAF: Multimodal Split Attention Fusion

Abstract

Multimodal learning mimics the reasoning process of the human multi-sensory system, which is used to perceive the surrounding world. While making a prediction, the human brain tends to relate crucial cues from multiple sources of information. In this work, we propose a novel multimodal fusion module that learns to emphasize more contributive features across all modalities. Specifically, the proposed Multimodal Split Attention Fusion (MSAF) module splits each modality into channel-wise equal feature blocks and creates a joint representation that is used to generate soft attention for each channel across the feature blocks. Further, the MSAF module is designed to be compatible with features of various spatial dimensions and sequence lengths, suitable for both CNNs and RNNs. Thus, MSAF can be easily added to fuse features of any unimodal networks and utilize existing pretrained unimodal model weights. To demonstrate the effectiveness of our fusion module, we design three multimodal networks with MSAF for emotion recognition, sentiment analysis, and action recognition tasks. Our approach achieves competitive results in each task and outperforms other application-specific networks and multimodal fusion benchmarks.

References in corpus (9)

• DropBlock: A regularization method for convolutional networks
• MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
• Learning Factorized Multimodal Representations
• Speech Emotion Recognition with Dual-Sequence LSTM Architecture
• Learn to Combine Modalities in Multimodal Deep Learning
• Infrared and 3D skeleton feature fusion for RGB-D action recognition
• Multimodal Fusion with Deep Neural Networks for Audio-Video Emotion Recognition
• Feedback Graph Convolutional Network for Skeleton-based Action Recognition
• A Spontaneous Driver Emotion Facial Expression (DEFE) Dataset for Intelligent Vehicles

Cited by in corpus (4)

• HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition
• SVFAP: Self-supervised Video Facial Affect Perceiver
• A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition
• MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition