Improving Multimodal fusion via Mutual Dependency Maximisation

Abstract

Multimodal sentiment analysis is a trending area of research, and the multimodal fusion is one of its most active topic. Acknowledging humans communicate through a variety of channels (i.e visual, acoustic, linguistic), multimodal systems aim at integrating different unimodal representations into a synthetic one. So far, a consequent effort has been made on developing complex architectures allowing the fusion of these modalities. However, such systems are mainly trained by minimising simple losses such as $L_1$ or cross-entropy. In this work, we investigate unexplored penalties and propose a set of new objectives that measure the dependency between modalities. We demonstrate that our new penalties lead to a consistent improvement (up to $4.3$ on accuracy) across a large variety of state-of-the-art models on two well-known sentiment analysis datasets: \texttt{CMU-MOSI} and \texttt{CMU-MOSEI}. Our method not only achieves a new SOTA on both datasets but also produces representations that are more robust to modality drops. Finally, a by-product of our methods includes a statistical network which can be used to interpret the high dimensional representations learnt by the model.

References in corpus (15)

• Decoupled Weight Decay Regularization
• Deep Learning using Rectified Linear Units (ReLU)
• An Overview of Multi-Task Learning in Deep Neural Networks
• Empirical Evaluation of Rectified Activations in Convolutional Network
• A Survey on Multi-view Learning
• Equitability, mutual information, and the maximal information coefficient
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Contrastive Multiview Coding
• Memory Fusion Network for Multi-view Sequential Learning
• Multivariate Dependence Beyond Shannon Information
• Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
• Factorized Multimodal Transformer for Multimodal Sequential Learning
• Code-switched inspired losses for generic spoken dialog representations
• A multimodal movie review corpus for fine-grained opinion mining
• Affine-Invariant Integrated Rank-Weighted Depth: Definition, Properties and Finite Sample Analysis