Smooth Neighbors on Teacher Graphs for Semi-supervised Learning

Abstract

The recently proposed self-ensembling methods have achieved promising results in deep semi-supervised learning, which penalize inconsistent predictions of unlabeled data under different perturbations. However, they only consider adding perturbations to each single data point, while ignoring the connections between data samples. In this paper, we propose a novel method, called Smooth Neighbors on Teacher Graphs (SNTG). In SNTG, a graph is constructed based on the predictions of the teacher model, i.e., the implicit self-ensemble of models. Then the graph serves as a similarity measure with respect to which the representations of "similar" neighboring points are learned to be smooth on the low-dimensional manifold. We achieve state-of-the-art results on semi-supervised learning benchmarks. The error rates are 9.89%, 3.99% for CIFAR-10 with 4000 labels, SVHN with 500 labels, respectively. In particular, the improvements are significant when the labels are fewer. For the non-augmented MNIST with only 20 labels, the error rate is reduced from previous 4.81% to 1.36%. Our method also shows robustness to noisy labels.

arXiv commentAccept as Spotlight in Computer Vision and Pattern Recognition 2018

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Improving neural networks by preventing co-adaptation of feature detectors
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Temporal Ensembling for Semi-Supervised Learning
• Improved Techniques for Training GANs
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• Adversarially Learned Inference
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
• Learning with Pseudo-Ensembles
• Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks
• Good Semi-supervised Learning that Requires a Bad GAN
• Triple Generative Adversarial Nets
• Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
• Unifying distillation and privileged information
• Semi-supervised Learning with GANs: Manifold Invariance with Improved Inference

Cited by in corpus (5)

• Contrastively Smoothed Class Alignment for Unsupervised Domain Adaptation
• Unsupervised Domain Adaptation using Generative Models and Self-ensembling
• Gradient-based Data Augmentation for Semi-Supervised Learning
• Patch-level Neighborhood Interpolation: A General and Effective Graph-based Regularization Strategy
• Pairwise Teacher-Student Network for Semi-Supervised Hashing