Repetitive Reprediction Deep Decipher for Semi-Supervised Learning

Abstract

Most recent semi-supervised deep learning (deep SSL) methods used a similar paradigm: use network predictions to update pseudo-labels and use pseudo-labels to update network parameters iteratively. However, they lack theoretical support and cannot explain why predictions are good candidates for pseudo-labels. In this paper, we propose a principled end-to-end framework named deep decipher (D2) for SSL. Within the D2 framework, we prove that pseudo-labels are related to network predictions by an exponential link function, which gives a theoretical support for using predictions as pseudo-labels. Furthermore, we demonstrate that updating pseudo-labels by network predictions will make them uncertain. To mitigate this problem, we propose a training strategy called repetitive reprediction (R2). Finally, the proposed R2-D2 method is tested on the large-scale ImageNet dataset and outperforms state-of-the-art methods by 5 percentage points.

arXiv commentAccepted by AAAI 2020

References in corpus (6)

• Improved Regularization of Convolutional Neural Networks with Cutout
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Deep Label Distribution Learning with Label Ambiguity
• Variational Autoencoder for Deep Learning of Images, Labels and Captions
• Shake-Shake regularization

Cited by in corpus (2)

• Neural Network Pruning with Residual-Connections and Limited-Data
• SSMD: Semi-Supervised Medical Image Detection with Adaptive Consistency and Heterogeneous Perturbation