Towards Balanced Active Learning for Multimodal Classification

Abstract

Training multimodal networks requires a vast amount of data due to their larger parameter space compared to unimodal networks. Active learning is a widely used technique for reducing data annotation costs by selecting only those samples that could contribute to improving model performance. However, current active learning strategies are mostly designed for unimodal tasks, and when applied to multimodal data, they often result in biased sample selection from the dominant modality. This unfairness hinders balanced multimodal learning, which is crucial for achieving optimal performance. To address this issue, we propose three guidelines for designing a more balanced multimodal active learning strategy. Following these guidelines, a novel approach is proposed to achieve more fair data selection by modulating the gradient embedding with the dominance degree among modalities. Our studies demonstrate that the proposed method achieves more balanced multimodal learning by avoiding greedy sample selection from the dominant modality. Our approach outperforms existing active learning strategies on a variety of multimodal classification tasks. Overall, our work highlights the importance of balancing sample selection in multimodal active learning and provides a practical solution for achieving more balanced active learning for multimodal classification.

arXiv comment12 pages, accepted by ACMMM 2023

References in corpus (11)

• Decoupled Weight Decay Regularization
• The Kinetics Human Action Video Dataset
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Active Learning for Convolutional Neural Networks: A Core-Set Approach
• Deep Bayesian Active Learning with Image Data
• Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds
• Adversarial Active Learning for Deep Networks: a Margin Based Approach
• Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets
• Batch Active Learning Using Determinantal Point Processes
• Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)
• Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks

Cited by in corpus (1)

• Enhancing Modality Representation and Alignment for Multimodal Cold-start Active Learning