What Makes Multi-modal Learning Better than Single (Provably)
arXiv:2106.04538
Abstract
The world provides us with data of multiple modalities. Intuitively, models fusing data from different modalities outperform their uni-modal counterparts, since more information is aggregated. Recently, joining the success of deep learning, there is an influential line of work on deep multi-modal learning, which has remarkable empirical results on various applications. However, theoretical justifications in this field are notably lacking. Can multi-modal learning provably perform better than uni-modal? In this paper, we answer this question under a most popular multi-modal fusion framework, which firstly encodes features from different modalities into a common latent space and seamlessly maps the latent representations into the task space. We prove that learning with multiple modalities achieves a smaller population risk than only using its subset of modalities. The main intuition is that the former has a more accurate estimate of the latent space representation. To the best of our knowledge, this is the first theoretical treatment to capture important qualitative phenomena observed in real multi-modal applications from the generalization perspective. Combining with experiment results, we show that multi-modal learning does possess an appealing formal guarantee.
arXiv commentAccepted to NeurIPS 2021
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Learning Transferable Visual Models From Natural Language Supervision
- A Survey on Multi-view Learning
- Deep Continuous Fusion for Multi-Sensor 3D Object Detection
- Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
- RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentation
- Spectrally-normalized margin bounds for neural networks
- Deep Multimodal Fusion by Channel Exchanging
- Few-Shot Learning via Learning the Representation, Provably
- Provable Meta-Learning of Linear Representations
- Improving Multi-Modal Learning with Uni-Modal Teachers
- Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies