Efficient Low-rank Multimodal Fusion with Modality-Specific Factors

Abstract

Multimodal research is an emerging field of artificial intelligence, and one of the main research problems in this field is multimodal fusion. The fusion of multimodal data is the process of integrating multiple unimodal representations into one compact multimodal representation. Previous research in this field has exploited the expressiveness of tensors for multimodal representation. However, these methods often suffer from exponential increase in dimensions and in computational complexity introduced by transformation of input into tensor. In this paper, we propose the Low-rank Multimodal Fusion method, which performs multimodal fusion using low-rank tensors to improve efficiency. We evaluate our model on three different tasks: multimodal sentiment analysis, speaker trait analysis, and emotion recognition. Our model achieves competitive results on all these tasks while drastically reducing computational complexity. Additional experiments also show that our model can perform robustly for a wide range of low-rank settings, and is indeed much more efficient in both training and inference compared to other methods that utilize tensor representations.

arXiv comment* Equal contribution. 10 pages. Accepted by ACL 2018

References in corpus (3)

• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
• Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysis

Cited by in corpus (18)

• Learning Factorized Multimodal Representations
• UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
• Multimodal Speech Emotion Recognition and Ambiguity Resolution
• Factorized Multimodal Transformer for Multimodal Sequential Learning
• Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis
• Improving Multimodal Accuracy Through Modality Pre-training and Attention
• SMIL: Multimodal Learning with Severely Missing Modality
• Integrating Multimodal Information in Large Pretrained Transformers
• Seq2Tens: An Efficient Representation of Sequences by Low-Rank Tensor Projections
• A novel multimodal approach for hybrid brain-computer interface
• H-OWAN: Multi-distorted Image Restoration with Tensor 1x1 Convolution
• TEASEL: A Transformer-Based Speech-Prefixed Language Model
• Towards A Multi-agent System for Online Hate Speech Detection
• Low-Rank HOCA: Efficient High-Order Cross-Modal Attention for Video Captioning
• A Multimodal Sentiment Dataset for Video Recommendation
• Neural Dependency Coding inspired Multimodal Fusion
• TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis
• Sentence-level Online Handwritten Chinese Character Recognition