Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning

Abstract

We propose a new regularization method based on virtual adversarial loss: a new measure of local smoothness of the conditional label distribution given input. Virtual adversarial loss is defined as the robustness of the conditional label distribution around each input data point against local perturbation. Unlike adversarial training, our method defines the adversarial direction without label information and is hence applicable to semi-supervised learning. Because the directions in which we smooth the model are only "virtually" adversarial, we call our method virtual adversarial training (VAT). The computational cost of VAT is relatively low. For neural networks, the approximated gradient of virtual adversarial loss can be computed with no more than two pairs of forward- and back-propagations. In our experiments, we applied VAT to supervised and semi-supervised learning tasks on multiple benchmark datasets. With a simple enhancement of the algorithm based on the entropy minimization principle, our VAT achieves state-of-the-art performance for semi-supervised learning tasks on SVHN and CIFAR-10.

To be appeared in IEEE Transactions on Pattern Analysis and Machine Intelligence

References in corpus (2)

• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• A Bayesian encourages dropout

Cited by in corpus (63)

• Machine learning methods for histopathological image analysis
• AutoAugment: Learning Augmentation Policies from Data
• A DIRT-T Approach to Unsupervised Domain Adaptation
• Rethinking Pre-training and Self-training
• Robustness May Be at Odds with Accuracy
• Scalable Private Learning with PATE
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Good Semi-supervised Learning that Requires a Bad GAN
• PseudoSeg: Designing Pseudo Labels for Semantic Segmentation
• Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
• Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Training
• Do Adversarially Robust ImageNet Models Transfer Better?
• A Survey on Text Classification: From Shallow to Deep Learning
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• A Semantic Loss Function for Deep Learning with Symbolic Knowledge
• Sorting out Lipschitz function approximation
• Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision
• Self-ensembling for visual domain adaptation
• Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect
• Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance
• Deep Co-Training for Semi-Supervised Image Recognition
• GAR: An efficient and scalable Graph-based Activity Regularization for semi-supervised learning
• Smooth Neighbors on Teacher Graphs for Semi-supervised Learning
• Machine Learning with Membership Privacy using Adversarial Regularization
• Intriguing Properties of Adversarial Examples
• ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation
• Fooling a Real Car with Adversarial Traffic Signs
• Semi-Supervised Sequence Modeling with Cross-View Training
• RDEC: Integrating Regularization into Deep Embedded Clustering for Imbalanced Datasets
• Adversarially Robust Training through Structured Gradient Regularization
• Projection Based Weight Normalization for Deep Neural Networks
• Manifold regularization with GANs for semi-supervised learning
• Deep Metric Transfer for Label Propagation with Limited Annotated Data
• Improving Network Robustness against Adversarial Attacks with Compact Convolution
• Gradient Band-based Adversarial Training for Generalized Attack Immunity of A3C Path Finding
• A Semi-Supervised Two-Stage Approach to Learning from Noisy Labels
• Neural Semi-supervised Learning for Text Classification Under Large-Scale Pretraining
• Leveraging Unlabelled Data in Multiple-Instance Learning Problems for Improved Detection of Parkinsonian Tremor in Free-Living Conditions
• What If We Only Use Real Datasets for Scene Text Recognition? Toward Scene Text Recognition With Fewer Labels
• Negative sampling in semi-supervised learning
• Manifold Adversarial Learning
• Virtual Adversarial Ladder Networks For Semi-supervised Learning
• AutoGAN: Robust Classifier Against Adversarial Attacks
• Learning Optimal Data Augmentation Policies via Bayesian Optimization for Image Classification Tasks
• Adversarial Constraint Learning for Structured Prediction
• Improving Model Robustness with Latent Distribution Locally and Globally
• SaaS: Speed as a Supervisor for Semi-supervised Learning
• FineFool: Fine Object Contour Attack via Attention
• Semi-supervised learning for medical image classification using imbalanced training data
• Adversarially Learned Mixture Model
• On Norm-Agnostic Robustness of Adversarial Training
• Semi-Supervised Learning with IPM-based GANs: an Empirical Study
• Constrained Instance and Class Reweighting for Robust Learning under Label Noise
• ROI Regularization for Semi-supervised and Supervised Learning
• Gradient-based Data Augmentation for Semi-Supervised Learning
• Adversarial Dropout for Recurrent Neural Networks
• Semi-supervised Learning with Contrastive Predicative Coding
• Adversarial confidence and smoothness regularizations for scalable unsupervised discriminative learning
• Semi-supervised Acoustic Event Detection based on tri-training
• Combining Diverse Feature Priors
• Semi-Supervised Learning for Lensed Quasar Detection
• TEAM: We Need More Powerful Adversarial Examples for DNNs
• Iterative Self-Learning: Semi-Supervised Improvement to Dataset Volumes and Model Accuracy