EmbraceNet: A robust deep learning architecture for multimodal classification

Abstract

Classification using multimodal data arises in many machine learning applications. It is crucial not only to model cross-modal relationship effectively but also to ensure robustness against loss of part of data or modalities. In this paper, we propose a novel deep learning-based multimodal fusion architecture for classification tasks, which guarantees compatibility with any kind of learning models, deals with cross-modal information carefully, and prevents performance degradation due to partial absence of data. We employ two datasets for multimodal classification tasks, build models based on our architecture and other state-of-the-art models, and analyze their performance on various situations. The results show that our architecture outperforms the other multimodal fusion architectures when some parts of data are not available.

Code available at https://github.com/idearibosome/embracenet

References in corpus (3)

• Deep Learning in Neural Networks: An Overview
• YFCC100M: The New Data in Multimedia Research
• Perceptual Experience Analysis for Tone-mapped HDR Videos based on EEG and Peripheral Physiological Signals

Cited by in corpus (10)

• Human Activity Recognition Using Tools of Convolutional Neural Networks: A State of the Art Review, Data Sets, Challenges and Future Prospects
• Imbalanced Big Data Oversampling: Taxonomy, Algorithms, Software, Guidelines and Future Directions
• Data fusion strategies for energy efficiency in buildings: Overview, challenges and novel orientations
• Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation
• DANA: Dimension-Adaptive Neural Architecture for Multivariate Sensor Data
• EmbraceNet for Activity: A Deep Multimodal Fusion Architecture for Activity Recognition
• Whose Emotion Matters? Speaking Activity Localisation without Prior Knowledge
• Missing Data as Augmentation in the Earth Observation Domain: A Multi-View Learning Approach
• A Light-weight Deep Human Activity Recognition Algorithm Using Multi-knowledge Distillation
• Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration