Distributional Smoothing with Virtual Adversarial Training

Abstract

We propose local distributional smoothness (LDS), a new notion of smoothness for statistical model that can be used as a regularization term to promote the smoothness of the model distribution. We named the LDS based regularization as virtual adversarial training (VAT). The LDS of a model at an input datapoint is defined as the KL-divergence based robustness of the model distribution against local perturbation around the datapoint. VAT resembles adversarial training, but distinguishes itself in that it determines the adversarial direction from the model distribution alone without using the label information, making it applicable to semi-supervised learning. The computational cost for VAT is relatively low. For neural network, the approximated gradient of the LDS can be computed with no more than three pairs of forward and back propagations. When we applied our technique to supervised and semi-supervised learning for the MNIST dataset, it outperformed all the training methods other than the current state of the art method, which is based on a highly advanced generative model. We also applied our method to SVHN and NORB, and confirmed our method's superior performance over the current state of the art semi-supervised method applied to these datasets.

Under review as a conference paper at ICLR 2016

References in corpus (5)

• Improving neural networks by preventing co-adaptation of feature detectors
• Theano: new features and speed improvements
• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• Learning with Pseudo-Ensembles
• Lateral Connections in Denoising Autoencoders Support Supervised Learning

Cited by in corpus (96)

• Deep Bayesian Active Learning with Image Data
• Certified Defenses against Adversarial Examples
• A Survey on Data Augmentation for Text Classification
• Deep Learning Based Text Classification: A Comprehensive Review
• Understanding Adversarial Training: Increasing Local Stability of Neural Nets through Robust Optimization
• Certifying Some Distributional Robustness with Principled Adversarial Training
• Learning Discrete Representations via Information Maximizing Self-Augmented Training
• Pathologies of Neural Models Make Interpretations Difficult
• Auxiliary Deep Generative Models
• Learning with a Strong Adversary
• Good Semi-supervised Learning that Requires a Bad GAN
• Triple Generative Adversarial Nets
• Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
• A General Framework for Adversarial Examples with Objectives
• Adversarial Training Methods for Semi-Supervised Text Classification
• Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities
• BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
• Semi-supervised Learning with GANs: Manifold Invariance with Improved Inference
• A Survey on Text Classification: From Shallow to Deep Learning
• Improving the Robustness of Deep Neural Networks via Stability Training
• Generative Adversarial Trainer: Defense to Adversarial Perturbations with GAN
• Adversarial Training against Location-Optimized Adversarial Patches
• Adversarially Robust Generalization Just Requires More Unlabeled Data
• Semi-supervised Domain Adaptation via Minimax Entropy
• A Semantic Loss Function for Deep Learning with Symbolic Knowledge
• HopSkipJumpAttack: A Query-Efficient Decision-Based Attack
• Salient Objects in Clutter
• Learning by Association - A versatile semi-supervised training method for neural networks
• Training individually fair ML models with Sensitive Subspace Robustness
• Adversarial Dropout Regularization
• On the Minimal Supervision for Training Any Binary Classifier from Only Unlabeled Data
• Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance
• Adversarial Deep Structural Networks for Mammographic Mass Segmentation
• SafetyNet: Detecting and Rejecting Adversarial Examples Robustly
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations
• Unsupervised Learning on Neural Network Outputs: with Application in Zero-shot Learning
• Interpretable Adversarial Perturbation in Input Embedding Space for Text
• Adversarial Training in Affective Computing and Sentiment Analysis: Recent Advances and Perspectives
• Confidence-Calibrated Adversarial Training: Generalizing to Unseen Attacks
• Mitigating Overfitting in Supervised Classification from Two Unlabeled Datasets: A Consistent Risk Correction Approach
• A Theoretical Framework for Robustness of (Deep) Classifiers against Adversarial Examples
• Machine Learning with Membership Privacy using Adversarial Regularization
• OpenMatch: Open-set Consistency Regularization for Semi-supervised Learning with Outliers
• Global Robustness Evaluation of Deep Neural Networks with Provable Guarantees for the $L_0$ Norm
• A Baseline for Few-Shot Image Classification
• Attacks on State-of-the-Art Face Recognition using Attentional Adversarial Attack Generative Network
• The Attacker's Perspective on Automatic Speaker Verification: An Overview
• COPYCAT: Practical Adversarial Attacks on Visualization-Based Malware Detection
• HashTran-DNN: A Framework for Enhancing Robustness of Deep Neural Networks against Adversarial Malware Samples
• Semi-Supervised Sequence Modeling with Cross-View Training
• Speech Recognition: Keyword Spotting Through Image Recognition
• Towards Robust Neural Machine Translation
• Denoising Distant Supervision for Relation Extraction via Instance-Level Adversarial Training
• Unifying Adversarial Training Algorithms with Flexible Deep Data Gradient Regularization
• Complementary-Label Learning for Arbitrary Losses and Models
• Adversarially Robust Training through Structured Gradient Regularization
• Neural Simpletrons - Minimalistic Directed Generative Networks for Learning with Few Labels
• Ensemble Robustness and Generalization of Stochastic Deep Learning Algorithms
• Generalization in Machine Learning via Analytical Learning Theory
• Semi-Supervised Generation with Cluster-aware Generative Models
• Exploring Self-Supervised Regularization for Supervised and Semi-Supervised Learning
• Incorporating Unlabeled Data into Distributionally Robust Learning
• Adversarial Binaries for Authorship Identification
• Adversarial Robustness Assessment: Why both $L_0$ and $L_{\infty }$ Attacks Are Necessary
• Triple Generative Adversarial Networks
• Bit Error Robustness for Energy-Efficient DNN Accelerators
• Improving ClusterGAN Using Self-Augmented Information Maximization of Disentangling Latent Spaces
• Deep-learning Real/Bogus classification for the Tomo-e Gozen transient survey
• Boundary-Refined Prototype Generation: A General End-to-End Paradigm for Semi-Supervised Semantic Segmentation
• Rethinking Curriculum Learning with Incremental Labels and Adaptive Compensation
• Adversarial Detection: Attacking Object Detection in Real Time
• The Adversarial Attack and Detection under the Fisher Information Metric
• Manifold Adversarial Learning
• Semi-Supervised and Active Few-Shot Learning with Prototypical Networks
• DRo: A data-scarce mechanism to revolutionize the performance of Deep Learning based Security Systems
• On the Estimation of Information Measures of Continuous Distributions
• Semi-Supervised Learning Enabled by Multiscale Deep Neural Network Inversion
• A Biologically Inspired Visual Working Memory for Deep Networks
• One Bit Matters: Understanding Adversarial Examples as the Abuse of Redundancy
• A Game Theoretic Analysis of Additive Adversarial Attacks and Defenses
• Infomax Neural Joint Source-Channel Coding via Adversarial Bit Flip
• Attentive Representation Learning with Adversarial Training for Short Text Clustering
• Understanding and Improving Virtual Adversarial Training
• Annotation Cost Reduction of Stream-based Active Learning by Automated Weak Labeling using a Robot Arm
• Ensemble Manifold Segmentation for Model Distillation and Semi-supervised Learning
• SaaS: Speed as a Supervisor for Semi-supervised Learning
• Iterative Window Mean Filter: Thwarting Diffusion-based Adversarial Purification
• HybridNet: Classification and Reconstruction Cooperation for Semi-Supervised Learning
• Single-Solution Hypervolume Maximization and its use for Improving Generalization of Neural Networks
• Distributional Robust Kelly Gambling: Optimal Strategy under Uncertainty in the Long-Run
• Can audio-visual integration strengthen robustness under multimodal attacks?
• Representation Quality Of Neural Networks Links To Adversarial Attacks and Defences
• Multi-Expert Adversarial Attack Detection in Person Re-identification Using Context Inconsistency
• SSLayout360: Semi-Supervised Indoor Layout Estimation from 360-Degree Panorama
• Regularization with Latent Space Virtual Adversarial Training
• Semi-Supervised Self-Growing Generative Adversarial Networks for Image Recognition