Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free

Abstract

Adversarial training and its many variants substantially improve deep network robustness, yet at the cost of compromising standard accuracy. Moreover, the training process is heavy and hence it becomes impractical to thoroughly explore the trade-off between accuracy and robustness. This paper asks this new question: how to quickly calibrate a trained model in-situ, to examine the achievable trade-offs between its standard and robust accuracies, without (re-)training it many times? Our proposed framework, Once-for-all Adversarial Training (OAT), is built on an innovative model-conditional training framework, with a controlling hyper-parameter as the input. The trained model could be adjusted among different standard and robust accuracies "for free" at testing time. As an important knob, we exploit dual batch normalization to separate standard and adversarial feature statistics, so that they can be learned in one model without degrading performance. We further extend OAT to a Once-for-all Adversarial Training and Slimming (OATS) framework, that allows for the joint trade-off among accuracy, robustness and runtime efficiency. Experiments show that, without any re-training nor ensembling, OAT/OATS achieve similar or even superior performance compared to dedicatedly trained models at various configurations. Our codes and pretrained models are available at: https://github.com/VITA-Group/Once-for-All-Adversarial-Training.

arXiv commentNeurIPS 2020

References in corpus (13)

• Conditional Generative Adversarial Nets
• End to End Learning for Self-Driving Cars
• Wide Residual Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Adversarial Training Can Hurt Generalization
• MMA Training: Direct Input Space Margin Maximization through Adversarial Training
• Adversarial Robustness May Be at Odds With Simplicity
• Model Compression with Adversarial Robustness: A Unified Optimization Framework
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference
• On the Connection Between Adversarial Robustness and Saliency Map Interpretability
• Adjustable Real-time Style Transfer
• Towards Understanding Adversarial Examples Systematically: Exploring Data Size, Task and Model Factors

Cited by in corpus (11)

• Advancing diagnostic performance and clinical usability of neural networks via adversarial training and dual batch normalization
• Geometry-aware Instance-reweighted Adversarial Training
• Probabilistic Margins for Instance Reweighting in Adversarial Training
• Reliable Adversarial Distillation with Unreliable Teachers
• Generative Dynamic Patch Attack
• Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
• Adversarial Robustness with Non-uniform Perturbations
• Learning Diverse-Structured Networks for Adversarial Robustness
• Push Stricter to Decide Better: A Class-Conditional Feature Adaptive Framework for Improving Adversarial Robustness
• Closing the Gap: Achieving Better Accuracy-Robustness Tradeoffs against Query-Based Attacks
• Pareto Adversarial Robustness: Balancing Spatial Robustness and Sensitivity-based Robustness