SaaS: Speed as a Supervisor for Semi-supervised Learning
arXiv:1805.00980
Abstract
We introduce the SaaS Algorithm for semi-supervised learning, which uses learning speed during stochastic gradient descent in a deep neural network to measure the quality of an iterative estimate of the posterior probability of unknown labels. Training speed in supervised learning correlates strongly with the percentage of correct labels, so we use it as an inference criterion for the unknown labels, without attempting to infer the model parameters at first. Despite its simplicity, SaaS achieves state-of-the-art results in semi-supervised learning benchmarks.
References in corpus (14)
- Semi-Supervised Classification with Graph Convolutional Networks
- Temporal Ensembling for Semi-Supervised Learning
- Understanding deep learning requires rethinking generalization
- Adversarially Learned Inference
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
- Distributional Smoothing with Virtual Adversarial Training
- Train faster, generalize better: Stability of stochastic gradient descent
- Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks
- Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
- Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis
- Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
- A Semantic Loss Function for Deep Learning with Symbolic Knowledge
- Learning by Association - A versatile semi-supervised training method for neural networks