Multi-stage Optimization based Adversarial Training

Abstract

In the field of adversarial robustness, there is a common practice that adopts the single-step adversarial training for quickly developing adversarially robust models. However, the single-step adversarial training is most likely to cause catastrophic overfitting, as after a few training epochs it will be hard to generate strong adversarial examples to continuously boost the adversarial robustness. In this work, we aim to avoid the catastrophic overfitting by introducing multi-step adversarial examples during the single-step adversarial training. Then, to balance the large training overhead of generating multi-step adversarial examples, we propose a Multi-stage Optimization based Adversarial Training (MOAT) method that periodically trains the model on mixed benign examples, single-step adversarial examples, and multi-step adversarial examples stage by stage. In this way, the overall training overhead is reduced significantly, meanwhile, the model could avoid catastrophic overfitting. Extensive experiments on CIFAR-10 and CIFAR-100 datasets demonstrate that under similar amount of training overhead, the proposed MOAT exhibits better robustness than either single-step or multi-step adversarial training methods.

arXiv comment13 pages

References in corpus (10)

• mixup: Beyond Empirical Risk Minimization
• Delving into Transferable Adversarial Examples and Black-box Attacks
• Certified Adversarial Robustness via Randomized Smoothing
• Fast is better than free: Revisiting adversarial training
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Semidefinite relaxations for certifying robustness to adversarial examples
• Adversarial Attacks Against Medical Deep Learning Systems
• MMA Training: Direct Input Space Margin Maximization through Adversarial Training
• Towards Understanding Fast Adversarial Training
• Understanding Catastrophic Overfitting in Single-step Adversarial Training