Semi-Supervised Learning with Meta-Gradient

Abstract

In this work, we propose a simple yet effective meta-learning algorithm in semi-supervised learning. We notice that most existing consistency-based approaches suffer from overfitting and limited model generalization ability, especially when training with only a small number of labeled data. To alleviate this issue, we propose a learn-to-generalize regularization term by utilizing the label information and optimize the problem in a meta-learning fashion. Specifically, we seek the pseudo labels of the unlabeled data so that the model can generalize well on the labeled data, which is formulated as a nested optimization problem. We address this problem using the meta-gradient that bridges between the pseudo label and the regularization term. In addition, we introduce a simple first-order approximation to avoid computing higher-order derivatives and provide theoretic convergence analysis. Extensive evaluations on the SVHN, CIFAR, and ImageNet datasets demonstrate that the proposed algorithm performs favorably against state-of-the-art methods.

arXiv comment17 pages

References in corpus (10)

• Improving neural networks by preventing co-adaptation of feature detectors
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• DARTS: Differentiable Architecture Search
• Learning to Reweight Examples for Robust Deep Learning
• Semi-Supervised Learning with Ladder Networks
• Unrolled Generative Adversarial Networks
• Shake-Shake regularization
• Meta-Learning with Implicit Gradients
• There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average