Neural Dependency Coding inspired Multimodal Fusion

Abstract

Information integration from different modalities is an active area of research. Human beings and, in general, biological neural systems are quite adept at using a multitude of signals from different sensory perceptive fields to interact with the environment and each other. Recent work in deep fusion models via neural networks has led to substantial improvements over unimodal approaches in areas like speech recognition, emotion recognition and analysis, captioning and image description. However, such research has mostly focused on architectural changes allowing for fusion of different modalities while keeping the model complexity manageable. Inspired by recent neuroscience ideas about multisensory integration and processing, we investigate the effect of synergy maximizing loss functions. Experiments on multimodal sentiment analysis tasks: CMU-MOSI and CMU-MOSEI with different models show that our approach provides a consistent performance boost.

References in corpus (12)

• A Survey on Multi-view Learning
• Equitability, mutual information, and the maximal information coefficient
• Nonnegative Decomposition of Multivariate Information
• Quantifying High-order Interdependencies via Multivariate Extensions of the Mutual Information
• Memory Fusion Network for Multi-view Sequential Learning
• Intersection Information based on Common Randomness
• Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
• Multivariate Dependence Beyond Shannon Information
• Learning Deep Kernels for Non-Parametric Two-Sample Tests
• Factorized Multimodal Transformer for Multimodal Sequential Learning
• Understanding the Limitations of Variational Mutual Information Estimators
• A multimodal movie review corpus for fine-grained opinion mining