Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Too Much Accuracy

Abstract

Adversarial robustness has become a central goal in deep learning, both in the theory and the practice. However, successful methods to improve the adversarial robustness (such as adversarial training) greatly hurt generalization performance on the unperturbed data. This could have a major impact on how the adversarial robustness affects real world systems (i.e. many may opt to forego robustness if it can improve accuracy on the unperturbed data). We propose Interpolated Adversarial Training, which employs recently proposed interpolation based training methods in the framework of adversarial training. On CIFAR-10, adversarial training increases the standard test error (when there is no adversary) from 4.43% to 12.32%, whereas with our Interpolated adversarial training we retain the adversarial robustness while achieving a standard test error of only 6.45%. With our technique, the relative increase in the standard error for the robust model is reduced from 178.1% to just 45.5%. Moreover, we provide mathematical analysis of Interpolated Adversarial Training to confirm its efficiencies and demonstrate its advantages in terms of robustness and generalization.

arXiv commentThis is the latest version, which is published in the Journal, "Neural Networks", in 2022. All the previous results are unchanged. First two authors contributed equally

References in corpus (32)

• Explaining and Harnessing Adversarial Examples
• Intriguing properties of neural networks
• mixup: Beyond Empirical Risk Minimization
• Deep Residual Learning for Image Recognition
• Neural Architecture Search with Reinforcement Learning
• End to End Learning for Self-Driving Cars
• Wide Residual Networks
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Theoretically Principled Trade-off between Robustness and Accuracy
• Interpolation Consistency Training for Semi-Supervised Learning
• Opening the Black Box of Deep Neural Networks via Information
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Synthesizing Robust Adversarial Examples
• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Adversarially Robust Generalization Requires More Data
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Adversarial Machine Learning at Scale
• Robustness May Be at Odds with Accuracy
• Towards the Science of Security and Privacy in Machine Learning
• Spectrally-normalized margin bounds for neural networks
• Adversarial Training Can Hurt Generalization
• Scaling provable adversarial defenses
• Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Training
• Rademacher Complexity for Adversarially Robust Generalization
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Adversarially Robust Generalization Just Requires More Unlabeled Data
• Evaluating and Understanding the Robustness of Adversarial Logit Pairing
• Unlabeled Data Improves Adversarial Robustness
• Kernel and Rich Regimes in Overparametrized Models
• Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy
• Generalization Bounds for Vicinal Risk Minimization Principle

Cited by in corpus (4)

• Instance adaptive adversarial training: Improved accuracy tradeoffs in neural nets
• Understanding and Mitigating the Tradeoff Between Robustness and Accuracy
• Mixup Inference: Better Exploiting Mixup to Defend Adversarial Attacks
• CIFS: Improving Adversarial Robustness of CNNs via Channel-wise Importance-based Feature Selection