You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle

Abstract

Deep learning achieves state-of-the-art results in many tasks in computer vision and natural language processing. However, recent works have shown that deep networks can be vulnerable to adversarial perturbations, which raised a serious robustness issue of deep networks. Adversarial training, typically formulated as a robust optimization problem, is an effective way of improving the robustness of deep networks. A major drawback of existing adversarial training algorithms is the computational overhead of the generation of adversarial examples, typically far greater than that of the network training. This leads to the unbearable overall computational cost of adversarial training. In this paper, we show that adversarial training can be cast as a discrete time differential game. Through analyzing the Pontryagin's Maximal Principle (PMP) of the problem, we observe that the adversary update is only coupled with the parameters of the first layer of the network. This inspires us to restrict most of the forward and back propagation within the first layer of the network during adversary updates. This effectively reduces the total number of full forward and backward propagation to only one for each group of adversary updates. Therefore, we refer to this algorithm YOPO (You Only Propagate Once). Numerical experiments demonstrate that YOPO can achieve comparable defense accuracy with approximately 1/5 ~ 1/4 GPU time of the projected gradient descent (PGD) algorithm. Our codes are available at https://https://github.com/a1600012888/YOPO-You-Only-Propagate-Once.

arXiv commentAccepted as a conference paper at NeurIPS 2019

References in corpus (19)

• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Adversarial Attacks on Neural Networks for Graph Data
• Theoretically Principled Trade-off between Robustness and Accuracy
• PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• The Robust Manifold Defense: Adversarial Training using Generative Models
• Feature Denoising for Improving Adversarial Robustness
• Decoupled Parallel Backpropagation with Convergence Guarantee
• An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks
• L2-Nonexpansive Neural Networks
• Dynamically Unfolding Recurrent Restorer: A Moving Endpoint Control Method for Image Restoration
• Transport Analysis of Infinitely Deep Neural Network
• Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Training
• Enhancing the Robustness of Deep Neural Networks by Boundary Conditional GAN
• RANDOM MASK: Towards Robust Convolutional Neural Networks
• ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies
• Lifted Neural Networks
• Towards Robust ResNet: A Small Step but A Giant Leap

Cited by in corpus (35)

• Theoretically Principled Trade-off between Robustness and Accuracy
• A Review on Deep Learning in Medical Image Reconstruction
• A Review of Adversarial Attack and Defense for Classification Methods
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• Adversarially Robust Generalization Just Requires More Unlabeled Data
• DeepRobust: A PyTorch Library for Adversarial Attacks and Defenses
• Semantic Robustness of Models of Source Code
• Convergence of Adversarial Training in Overparametrized Neural Networks
• Overfitting in adversarially robust deep learning
• Dynamic Slicing for Deep Neural Networks
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• To be Robust or to be Fair: Towards Fairness in Adversarial Training
• Intriguing properties of adversarial training at scale
• Pontryagin Differentiable Programming: An End-to-End Learning and Control Framework
• LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Training
• Improving the Tightness of Convex Relaxation Bounds for Training Certifiably Robust Classifiers
• Adversarial Parameter Defense by Multi-Step Risk Minimization
• Rearchitecting Classification Frameworks For Increased Robustness
• HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization
• TESDA: Transform Enabled Statistical Detection of Attacks in Deep Neural Networks
• Understanding Catastrophic Overfitting in Adversarial Training
• Meta Adversarial Training against Universal Patches
• Amata: An Annealing Mechanism for Adversarial Training Acceleration
• A Differential Game Theoretic Neural Optimizer for Training Residual Networks
• Concurrent Adversarial Learning for Large-Batch Training
• ROBY: Evaluating the Robustness of a Deep Model by its Decision Boundaries
• Certified Defense via Latent Space Randomized Smoothing with Orthogonal Encoders
• Model-Agnostic Meta-Attack: Towards Reliable Evaluation of Adversarial Robustness
• The art of defense: letting networks fool the attacker
• Robust Deep Learning as Optimal Control: Insights and Convergence Guarantees
• Decoder-free Robustness Disentanglement without (Additional) Supervision
• Improving Resistance to Adversarial Deformations by Regularizing Gradients
• One Man's Trash is Another Man's Treasure: Resisting Adversarial Examples by Adversarial Examples
• Robustifying ${\ell }_{\infty }$ Adversarial Training to the Union of Perturbation Models
• Dynamic Game Theoretic Neural Optimizer