SMIL: Multimodal Learning with Severely Missing Modality

Abstract

A common assumption in multimodal learning is the completeness of training data, i.e., full modalities are available in all training examples. Although there exists research endeavor in developing novel methods to tackle the incompleteness of testing data, e.g., modalities are partially missing in testing examples, few of them can handle incomplete training modalities. The problem becomes even more challenging if considering the case of severely missing, e.g., 90% training examples may have incomplete modalities. For the first time in the literature, this paper formally studies multimodal learning with missing modality in terms of flexibility (missing modalities in training, testing, or both) and efficiency (most training data have incomplete modality). Technically, we propose a new method named SMIL that leverages Bayesian meta-learning in uniformly achieving both objectives. To validate our idea, we conduct a series of experiments on three popular benchmarks: MM-IMDb, CMU-MOSI, and avMNIST. The results prove the state-of-the-art performance of SMIL over existing methods and generative baselines including autoencoders and generative adversarial networks. Our code is available at https://github.com/mengmenm/SMIL.

In AAAI 2021 (9 pages)

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Prototypical Networks for Few-shot Learning
• MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
• Joint Multimodal Learning with Deep Generative Models
• Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
• Gated Multimodal Units for Information Fusion
• Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distribution Tasks
• Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledge

Cited by in corpus (7)

• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• Modality Fusion Network and Personalized Attention in Momentary Stress Detection in the Wild
• Maximum Likelihood Estimation for Multimodal Learning with Missing Modality
• Dizygotic Conditional Variational AutoEncoder for Multi-Modal and Partial Modality Absent Few-Shot Learning
• Towards dynamic multi-modal phenotyping using chest radiographs and physiological data
• Identifying Illicit Drug Dealers on Instagram with Large-scale Multimodal Data Fusion