What Makes Training Multi-Modal Classification Networks Hard?

Abstract

Consider end-to-end training of a multi-modal vs. a single-modal network on a task with multiple input modalities: the multi-modal network receives more information, so it should match or outperform its single-modal counterpart. In our experiments, however, we observe the opposite: the best single-modal network always outperforms the multi-modal network. This observation is consistent across different combinations of modalities and on different tasks and benchmarks. This paper identifies two main causes for this performance drop: first, multi-modal networks are often prone to overfitting due to increased capacity. Second, different modalities overfit and generalize at different rates, so training them jointly with a single optimization strategy is sub-optimal. We address these two problems with a technique we call Gradient Blending, which computes an optimal blend of modalities based on their overfitting behavior. We demonstrate that Gradient Blending outperforms widely-used baselines for avoiding overfitting and achieves state-of-the-art accuracy on various tasks including human action recognition, ego-centric action recognition, and acoustic event detection.

arXiv commentCVPR 2020

References in corpus (14)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• The Kinetics Human Action Video Dataset
• VQA: Visual Question Answering
• Zero-Shot Learning Through Cross-Modal Transfer
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• Spatiotemporal Residual Networks for Video Action Recognition
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• Object Level Visual Reasoning in Videos
• Multi-level Attention Model for Weakly Supervised Audio Classification
• Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification
• Gated Multimodal Units for Information Fusion
• SGD on Neural Networks Learns Functions of Increasing Complexity
• Efficient Large-Scale Multi-Modal Classification
• Baidu-UTS Submission to the EPIC-Kitchens Action Recognition Challenge 2019

Cited by in corpus (6)

• Supervised Multimodal Bitransformers for Classifying Images and Text
• Audiovisual SlowFast Networks for Video Recognition
• SemanticVoxels: Sequential Fusion for 3D Pedestrian Detection using LiDAR Point Cloud and Semantic Segmentation
• MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond
• Listen to Look: Action Recognition by Previewing Audio
• Data-efficient Alignment of Multimodal Sequences by Aligning Gradient Updates and Internal Feature Distributions