Improving Multimodal Accuracy Through Modality Pre-training and Attention
arXiv:2011.06102
Abstract
Training a multimodal network is challenging and it requires complex architectures to achieve reasonable performance. We show that one reason for this phenomena is the difference between the convergence rate of various modalities. We address this by pre-training modality-specific sub-networks in multimodal architectures independently before end-to-end training of the entire network. Furthermore, we show that the addition of an attention mechanism between sub-networks after pre-training helps identify the most important modality during ambiguous scenarios boosting the performance. We demonstrate that by performing these two tricks a simple network can achieve similar performance to a complicated architecture that is significantly more expensive to train on multiple tasks including sentiment analysis, emotion recognition, and speaker trait recognition.
References in corpus (11)
- Sequence to Sequence Learning with Neural Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- A Survey on Multi-view Learning
- Reasoning about Entailment with Neural Attention
- Understanding Neural Networks through Representation Erasure
- Learning Factorized Multimodal Representations
- Supervised Multimodal Bitransformers for Classifying Images and Text
- Memory Fusion Network for Multi-view Sequential Learning
- Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
- Seq2Seq2Sentiment: Multimodal Sequence to Sequence Models for Sentiment Analysis
- Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment