Learn to Combine Modalities in Multimodal Deep Learning

Abstract

Combining complementary information from multiple modalities is intuitively appealing for improving the performance of learning-based approaches. However, it is challenging to fully leverage different modalities due to practical challenges such as varying levels of noise and conflicts between modalities. Existing methods do not adopt a joint approach to capturing synergies between the modalities while simultaneously filtering noise and resolving conflicts on a per sample basis. In this work we propose a novel deep neural network based technique that multiplicatively combines information from different source modalities. Thus the model training process automatically focuses on information from more reliable modalities while reducing emphasis on the less reliable modalities. Furthermore, we propose an extension that multiplicatively combines not only the single-source modalities, but a set of mixtured source modalities to better capture cross-modal signal correlations. We demonstrate the effectiveness of our proposed technique by presenting empirical results on three multimodal classification tasks from different domains. The results show consistent accuracy improvements on all three tasks.

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Focal Loss for Dense Object Detection
• Searching for Exotic Particles in High-Energy Physics with Deep Learning
• Convolutional Neural Networks for Sentence Classification
• Recurrent Models of Visual Attention
• Multiple Object Recognition with Visual Attention
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Text
• Collaborative Layer-wise Discriminative Learning in Deep Neural Networks

Cited by in corpus (22)

• Machine Learning on Neutron and X-Ray Scattering
• Gas Detection and Identification Using Multimodal Artificial Intelligence Based Sensor Fusion
• Employing Multimodal Machine Learning for Stress Detection
• FPS-Net: A Convolutional Fusion Network for Large-Scale LiDAR Point Cloud Segmentation
• Integrating Chemical Language and Molecular Graph in Multimodal Fused Deep Learning for Drug Property Prediction
• MSAF: Multimodal Split Attention Fusion
• A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition
• Trajformer: Trajectory Prediction with Local Self-Attentive Contexts for Autonomous Driving
• Multi-modal Summarization for Video-containing Documents
• An Empirical Bayesian Approach to Limb-darkening in Modeling WASP-121b Transit Light Curves
• Deep Multimodal Feature Encoding for Video Ordering
• Multimodal Generative Models for Compositional Representation Learning
• Event and Activity Recognition in Video Surveillance for Cyber-Physical Systems
• Efficient Realistic Data Generation Framework leveraging Deep Learning-based Human Digitization
• 3D object quality prediction for Metal Jet Printer with Multimodal thermal encoder
• From Generalized zero-shot learning to long-tail with class descriptors
• Data Fusion for Deep Learning on Transport Mode Detection: A Case Study
• Towards Robust Pattern Recognition: A Review
• What is the appropriate speed for an autonomous vehicle? Designing a Pedestrian Aware Contextual Speed Controller
• A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection
• MIXER: A Principled Framework for Multimodal, Multiway Data Association
• Modal-aware Features for Multimodal Hashing