Amata: An Annealing Mechanism for Adversarial Training Acceleration
arXiv:2012.08112
Abstract
Despite the empirical success in various domains, it has been revealed that deep neural networks are vulnerable to maliciously perturbed input data that much degrade their performance. This is known as adversarial attacks. To counter adversarial attacks, adversarial training formulated as a form of robust optimization has been demonstrated to be effective. However, conducting adversarial training brings much computational overhead compared with standard training. In order to reduce the computational cost, we propose an annealing mechanism, Amata, to reduce the overhead associated with adversarial training. The proposed Amata is provably convergent, well-motivated from the lens of optimal control theory and can be combined with existing acceleration methods to further enhance performance. It is demonstrated that on standard datasets, Amata can achieve similar or better robustness with around 1/3 to 1/2 the computational time compared with traditional methods. In addition, Amata can be incorporated into other adversarial training acceleration algorithms (e.g. YOPO, Free, Fast, and ATTA), which leads to further reduction in computational time on large-scale problems.
References in corpus (10)
- Adam: A Method for Stochastic Optimization
- Fast is better than free: Revisiting adversarial training
- On the Convergence and Robustness of Adversarial Training
- Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey
- Attacks Which Do Not Kill Training Make Adversarial Learning Stronger
- Adversarial Example Defenses: Ensembles of Weak Defenses are not Strong
- You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
- Interpreting Adversarially Trained Convolutional Neural Networks
- Convergence of Adversarial Training in Overparametrized Neural Networks
- An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks