Collaborative Learning for Deep Neural Networks

Abstract

We introduce collaborative learning in which multiple classifier heads of the same network are simultaneously trained on the same training data to improve generalization and robustness to label noise with no extra inference cost. It acquires the strengths from auxiliary training, multi-task learning and knowledge distillation. There are two important mechanisms involved in collaborative learning. First, the consensus of multiple views from different classifier heads on the same example provides supplementary information as well as regularization to each classifier, thereby improving generalization. Second, intermediate-level representation (ILR) sharing with backpropagation rescaling aggregates the gradient flows from all heads, which not only reduces training computational complexity, but also facilitates supervision to the shared layers. The empirical results on CIFAR and ImageNet datasets demonstrate that deep neural networks learned as a group in a collaborative way significantly reduce the generalization error and increase the robustness to label noise.

arXiv commentTo appear in NIPS 2018

Cited by in corpus (11)

• On the Effects of Different Types of Label Noise in Multi-Label Remote Sensing Image Classification
• Challenges of Privacy-Preserving Machine Learning in IoT
• Towards Better Accuracy-efficiency Trade-offs: Divide and Co-training
• Feature Fusion for Online Mutual Knowledge Distillation
• Improved Techniques for Training Adaptive Deep Networks
• HOPE: Hybrid-granularity Ordinal Prototype Learning for Progression Prediction of Mild Cognitive Impairment
• Collaborative Label Correction via Entropy Thresholding
• Online Knowledge Distillation with Diverse Peers
• Test-Time Adaptation for Nighttime Color-Thermal Semantic Segmentation
• Safeguarded Dynamic Label Regression for Generalized Noisy Supervision
• Extending Label Smoothing Regularization with Self-Knowledge Distillation