Local Clustering with Mean Teacher for Semi-supervised Learning

Abstract

The Mean Teacher (MT) model of Tarvainen and Valpola has shown favorable performance on several semi-supervised benchmark datasets. MT maintains a teacher model's weights as the exponential moving average of a student model's weights and minimizes the divergence between their probability predictions under diverse perturbations of the inputs. However, MT is known to suffer from confirmation bias, that is, reinforcing incorrect teacher model predictions. In this work, we propose a simple yet effective method called Local Clustering (LC) to mitigate the effect of confirmation bias. In MT, each data point is considered independent of other points during training; however, data points are likely to be close to each other in feature space if they share similar features. Motivated by this, we cluster data points locally by minimizing the pairwise distance between neighboring data points in feature space. Combined with a standard classification cross-entropy objective on labeled data points, the misclassified unlabeled data points are pulled towards high-density regions of their correct class with the help of their neighbors, thus improving model performance. We demonstrate on semi-supervised benchmark datasets SVHN and CIFAR-10 that adding our LC loss to MT yields significant improvements compared to MT and performance comparable to the state of the art in semi-supervised learning.

8 pages, 7 figures

References in corpus (12)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Temporal Ensembling for Semi-Supervised Learning
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Semi-Supervised Learning with Ladder Networks
• Auxiliary Deep Generative Models
• Neural network-based clustering using pairwise constraints
• Learning by Association - A versatile semi-supervised training method for neural networks
• Semi-Supervised Learning via Compact Latent Space Clustering
• Fast Gradient-Based Inference with Continuous Latent Variable Models in Auxiliary Form
• Notes on image annotation

Cited by in corpus (1)

• Consistency Regularization with Generative Adversarial Networks for Semi-Supervised Learning