Certainty Driven Consistency Loss on Multi-Teacher Networks for Semi-Supervised Learning

Abstract

One of the successful approaches in semi-supervised learning is based on the consistency regularization. Typically, a student model is trained to be consistent with teacher prediction for the inputs under different perturbations. To be successful, the prediction targets given by teacher should have good quality, otherwise the student can be misled by teacher. Unfortunately, existing methods do not assess the quality of the teacher targets. In this paper, we propose a novel Certainty-driven Consistency Loss (CCL) that exploits the predictive uncertainty in the consistency loss to let the student dynamically learn from reliable targets. Specifically, we propose two approaches, i.e. Filtering CCL and Temperature CCL to either filter out uncertain predictions or pay less attention on them in the consistency regularization. We further introduce a novel decoupled framework to encourage model difference. Experimental results on SVHN, CIFAR-10, and CIFAR-100 demonstrate the advantages of our method over a few existing methods.

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• On Calibration of Modern Neural Networks
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Learning with Pseudo-Ensembles
• There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average