Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions

Abstract

Multimodal deep learning systems which employ multiple modalities like text, image, audio, video, etc., are showing better performance in comparison with individual modalities (i.e., unimodal) systems. Multimodal machine learning involves multiple aspects: representation, translation, alignment, fusion, and co-learning. In the current state of multimodal machine learning, the assumptions are that all modalities are present, aligned, and noiseless during training and testing time. However, in real-world tasks, typically, it is observed that one or more modalities are missing, noisy, lacking annotated data, have unreliable labels, and are scarce in training or testing and or both. This challenge is addressed by a learning paradigm called multimodal co-learning. The modeling of a (resource-poor) modality is aided by exploiting knowledge from another (resource-rich) modality using transfer of knowledge between modalities, including their representations and predictive models. Co-learning being an emerging area, there are no dedicated reviews explicitly focusing on all challenges addressed by co-learning. To that end, in this work, we provide a comprehensive survey on the emerging area of multimodal co-learning that has not been explored in its entirety yet. We review implementations that overcome one or more co-learning challenges without explicitly considering them as co-learning challenges. We present the comprehensive taxonomy of multimodal co-learning based on the challenges addressed by co-learning and associated implementations. The various techniques employed to include the latest ones are reviewed along with some of the applications and datasets. Our final goal is to discuss challenges and perspectives along with the important ideas and directions for future work that we hope to be beneficial for the entire research community focusing on this exciting domain.

arXiv commentThis is published in Information Fusion Journal and published copy can be downloaded from https://authors.elsevier.com/c/1eIVS5a7-Gls0Y for 50 days. https://www.sciencedirect.com/science/article/pii/S1566253521002530

References in corpus (58)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• SMOTE: Synthetic Minority Over-sampling Technique
• Conditional Generative Adversarial Nets
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• A Simple Framework for Contrastive Learning of Visual Representations
• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• Language Models are Few-Shot Learners
• A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Domain Generalization: A Survey
• Zero-Shot Learning Through Cross-Modal Transfer
• Contrastive Multiview Coding
• MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Multi-Task Learning with Deep Neural Networks: A Survey
• MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• Billion-scale semi-supervised learning for image classification
• WebVision Database: Visual Learning and Understanding from Web Data
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation
• A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines
• An Overview of Deep Semi-Supervised Learning
• Learning Factorized Multimodal Representations
• Towards Automatic Learning of Procedures from Web Instructional Videos
• RUBi: Reducing Unimodal Biases in Visual Question Answering
• FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age
• Decoupling "when to update" from "how to update"
• Heterogeneous Domain Generalization via Domain Mixup
• Gas Detection and Identification Using Multimodal Artificial Intelligence Based Sensor Fusion
• A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis
• CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning
• Self-training Improves Pre-training for Natural Language Understanding
• Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
• Multimodal Transfer Deep Learning with Applications in Audio-Visual Recognition
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
• Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges
• Counterfactual VQA: A Cause-Effect Look at Language Bias
• Enriched Music Representations with Multiple Cross-modal Contrastive Learning
• Unsupervised Audio-Visual Subspace Alignment for High-Stakes Deception Detection
• Contrastive Visual-Linguistic Pretraining
• SMIL: Multimodal Learning with Severely Missing Modality
• MCQA: Multimodal Co-attention Based Network for Question Answering
• Question-Conditioned Counterfactual Image Generation for VQA
• Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models
• CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation
• Learning Alignment for Multimodal Emotion Recognition from Speech
• Zero-Shot Transfer VQA Dataset
• MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding
• Multimodal Co-Training for Selecting Good Examples from Webly Labeled Video
• COBRA: Contrastive Bi-Modal Representation Algorithm
• Audio-Visual Event Localization via Recursive Fusion by Joint Co-Attention
• MMED: A Multi-domain and Multi-modality Event Dataset
• Image Captioning with Unseen Objects

Cited by in corpus (5)

• Visuo-Haptic Object Perception for Robots: An Overview
• Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources
• From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscape
• Interpreting wealth distribution via poverty map inference using multimodal data
• Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration