Multimodal Knowledge Expansion
arXiv:2103.14431
Abstract
The popularity of multimodal sensors and the accessibility of the Internet have brought us a massive amount of unlabeled multimodal data. Since existing datasets and well-trained models are primarily unimodal, the modality gap between a unimodal network and unlabeled multimodal data poses an interesting problem: how to transfer a pre-trained unimodal network to perform the same task on unlabeled multimodal data? In this work, we propose multimodal knowledge expansion (MKE), a knowledge distillation-based framework to effectively utilize multimodal data without requiring labels. Opposite to traditional knowledge distillation, where the student is designed to be lightweight and inferior to the teacher, we observe that a multimodal student model consistently denoises pseudo labels and generalizes better than its teacher. Extensive experiments on four tasks and different modalities verify this finding. Furthermore, we connect the mechanism of MKE to semi-supervised learning and offer both empirical and theoretical explanations to understand the denoising capability of a multimodal student.
arXiv commentAccepted by ICCV 2021. Project website: https://tsinghua-mars-lab.github.io/MKE/
References in corpus (15)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- Rethinking Atrous Convolution for Semantic Image Segmentation
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
- FitNets: Hints for Thin Deep Nets
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
- End-to-End Multimodal Emotion Recognition using Deep Neural Networks
- MixMatch: A Holistic Approach to Semi-Supervised Learning
- Learning with Pseudo-Ensembles
- Billion-scale semi-supervised learning for image classification
- ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
- Deep Multimodal Fusion by Channel Exchanging
- Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
- What Makes Multi-modal Learning Better than Single (Provably)