Gradient-based Data Augmentation for Semi-Supervised Learning

Abstract

In semi-supervised learning (SSL), a technique called consistency regularization (CR) achieves high performance. It has been proved that the diversity of data used in CR is extremely important to obtain a model with high discrimination performance by CR. We propose a new data augmentation (Gradient-based Data Augmentation (GDA)) that is deterministically calculated from the image pixel value gradient of the posterior probability distribution that is the model output. We aim to secure effective data diversity for CR by utilizing three types of GDA. On the other hand, it has been demonstrated that the mixup method for labeled data and unlabeled data is also effective in SSL. We propose an SSL method named MixGDA by combining various mixup methods and GDA. The discrimination performance achieved by MixGDA is evaluated against the 13-layer CNN that is used as standard in SSL research. As a result, for CIFAR-10 (4000 labels), MixGDA achieves the same level of performance as the best performance ever achieved. For SVHN (250 labels, 500 labels and 1000 labels) and CIFAR-100 (10000 labels), MixGDA achieves state-of-the-art performance.

arXiv commentThe lower bound of the inequality (line 2 on page 6 ) changed to fit fact 1 (2). Typos in (9) corrected

References in corpus (14)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• mixup: Beyond Empirical Risk Minimization
• Improved Regularization of Convolutional Neural Networks with Cutout
• Wide Residual Networks
• Unsupervised Data Augmentation for Consistency Training
• Interpolation Consistency Training for Semi-Supervised Learning
• AutoAugment: Learning Augmentation Policies from Data
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Averaging Weights Leads to Wider Optima and Better Generalization
• Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
• There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
• Smooth Neighbors on Teacher Graphs for Semi-supervised Learning
• ROI Regularization for Semi-supervised and Supervised Learning