Smooth Adversarial Training

Abstract

It is commonly believed that networks cannot be both accurate and robust, that gaining robustness means losing accuracy. It is also generally believed that, unless making networks larger, network architectural elements would otherwise matter little in improving adversarial robustness. Here we present evidence to challenge these common beliefs by a careful study about adversarial training. Our key observation is that the widely-used ReLU activation function significantly weakens adversarial training due to its non-smooth nature. Hence we propose smooth adversarial training (SAT), in which we replace ReLU with its smooth approximations to strengthen adversarial training. The purpose of smooth activation functions in SAT is to allow it to find harder adversarial examples and compute better gradient updates during adversarial training. Compared to standard adversarial training, SAT improves adversarial robustness for "free", i.e., no drop in accuracy and no increase in computational cost. For example, without introducing additional computations, SAT significantly enhances ResNet-50's robustness from 33.0% to 42.3%, while also improving accuracy by 0.9% on ImageNet. SAT also works well with larger networks: it helps EfficientNet-L1 to achieve 82.2% accuracy and 58.6% robustness on ImageNet, outperforming the previous state-of-the-art defense by 9.5% for accuracy and 11.6% for robustness. Models are available at https://github.com/cihangxie/SmoothAdversarialTraining.

tech report

References in corpus (16)

• Neural Architecture Search with Reinforcement Learning
• Theoretically Principled Trade-off between Robustness and Accuracy
• Searching for Activation Functions
• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• On Evaluating Adversarial Robustness
• Fast is better than free: Revisiting adversarial training
• A study of the effect of JPG compression on adversarial images
• Improving Adversarial Robustness via Promoting Ensemble Diversity
• On the Convergence and Robustness of Adversarial Training
• Adversarial Robustness May Be at Odds With Simplicity
• Batch Normalization is a Cause of Adversarial Vulnerability
• E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles
• RANDOM MASK: Towards Robust Convolutional Neural Networks
• Smooth activations and reproducibility in deep networks
• TanhSoft -- a family of activation functions combining Tanh and Softplus
• Anti-Bandit Neural Architecture Search for Model Defense

Cited by in corpus (29)

• MedViT: A Robust Vision Transformer for Generalized Medical Image Classification
• Adversarial Example Detection for DNN Models: A Review and Experimental Comparison
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• Unsolved Problems in ML Safety
• RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
• Recent Advances in Adversarial Training for Adversarial Robustness
• An Effective Anti-Aliasing Approach for Residual Networks
• Robust Image Classification Using A Low-Pass Activation Function and DCT Augmentation
• Bag of Tricks for Adversarial Training
• SoK: Machine Learning Governance
• On the Generalization Properties of Adversarial Training
• Towards Robust Vision Transformer
• Adversarial Robustness under Long-Tailed Distribution
• Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better
• On Adversarial Robustness of 3D Point Cloud Classification under Adaptive Attacks
• Learning Diverse-Structured Networks for Adversarial Robustness
• Auditing AI models for Verified Deployment under Semantic Specifications
• Bridging the Gap Between Adversarial Robustness and Optimization Bias
• AdvFilter: Predictive Perturbation-aware Filtering against Adversarial Attack via Multi-domain Learning
• RoMA: Robust Model Adaptation for Offline Model-based Optimization
• Understanding and Achieving Efficient Robustness with Adversarial Supervised Contrastive Learning
• Lower Bounds on Cross-Entropy Loss in the Presence of Test-time Adversaries
• Deep Repulsive Prototypes for Adversarial Robustness
• Training Efficiency and Robustness in Deep Learning
• Understanding Robustness in Teacher-Student Setting: A New Perspective
• Identifying Layers Susceptible to Adversarial Attacks
• Countering Adversarial Examples: Combining Input Transformation and Noisy Training
• Less is More: Feature Selection for Adversarial Robustness with Compressive Counter-Adversarial Attacks
• Semantics-Preserving Adversarial Training