Smooth Adversarial Training

Abstract

It is commonly believed that networks cannot be both accurate and robust, that gaining robustness means losing accuracy. It is also generally believed that, unless making networks larger, network architectural elements would otherwise matter little in improving adversarial robustness. Here we present evidence to challenge these common beliefs by a careful study about adversarial training. Our key observation is that the widely-used ReLU activation function significantly weakens adversarial training due to its non-smooth nature. Hence we propose smooth adversarial training (SAT), in which we replace ReLU with its smooth approximations to strengthen adversarial training. The purpose of smooth activation functions in SAT is to allow it to find harder adversarial examples and compute better gradient updates during adversarial training. Compared to standard adversarial training, SAT improves adversarial robustness for "free", i.e., no drop in accuracy and no increase in computational cost. For example, without introducing additional computations, SAT significantly enhances ResNet-50's robustness from 33.0% to 42.3%, while also improving accuracy by 0.9% on ImageNet. SAT also works well with larger networks: it helps EfficientNet-L1 to achieve 82.2% accuracy and 58.6% robustness on ImageNet, outperforming the previous state-of-the-art defense by 9.5% for accuracy and 11.6% for robustness. Models are available at https://github.com/cihangxie/SmoothAdversarialTraining.

arXiv commenttech report

References in corpus (29)

• Neural Architecture Search with Reinforcement Learning
• Gaussian Error Linear Units (GELUs)
• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Theoretically Principled Trade-off between Robustness and Accuracy
• Searching for Activation Functions
• AutoAugment: Learning Augmentation Policies from Data
• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• On Evaluating Adversarial Robustness
• Fast is better than free: Revisiting adversarial training
• A study of the effect of JPG compression on adversarial images
• Towards the first adversarially robust neural network model on MNIST
• Averaging Weights Leads to Wider Optima and Better Generalization
• Improving Adversarial Robustness via Promoting Ensemble Diversity
• Adversarial Robustness through Local Linearization
• On the Convergence and Robustness of Adversarial Training
• Foveation-based Mechanisms Alleviate Adversarial Examples
• Continuously Differentiable Exponential Linear Units
• Adversarial Robustness May Be at Odds With Simplicity
• Ensemble Methods as a Defense to Adversarial Perturbations Against Deep Neural Networks
• Convergence of Adversarial Training in Overparametrized Neural Networks
• Batch Normalization is a Cause of Adversarial Vulnerability
• E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles
• RANDOM MASK: Towards Robust Convolutional Neural Networks
• Improved Adversarial Robustness by Reducing Open Space Risk via Tent Activations
• Protecting Neural Networks with Hierarchical Random Switching: Towards Better Robustness-Accuracy Trade-off for Stochastic Defenses
• Smooth activations and reproducibility in deep networks
• TanhSoft -- a family of activation functions combining Tanh and Softplus
• Anti-Bandit Neural Architecture Search for Model Defense

Cited by in corpus (30)

• MedViT: A Robust Vision Transformer for Generalized Medical Image Classification
• Adversarial Example Detection for DNN Models: A Review and Experimental Comparison
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• Unsolved Problems in ML Safety
• RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
• Recent Advances in Adversarial Training for Adversarial Robustness
• An Effective Anti-Aliasing Approach for Residual Networks
• Regional Homogeneity: Towards Learning Transferable Universal Adversarial Perturbations Against Defenses
• Robust Image Classification Using A Low-Pass Activation Function and DCT Augmentation
• Bag of Tricks for Adversarial Training
• SoK: Machine Learning Governance
• On the Generalization Properties of Adversarial Training
• Adversarial Robustness under Long-Tailed Distribution
• Towards Robust Vision Transformer
• Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better
• On Adversarial Robustness of 3D Point Cloud Classification under Adaptive Attacks
• Learning Diverse-Structured Networks for Adversarial Robustness
• Auditing AI models for Verified Deployment under Semantic Specifications
• Bridging the Gap Between Adversarial Robustness and Optimization Bias
• RoMA: Robust Model Adaptation for Offline Model-based Optimization
• Understanding and Achieving Efficient Robustness with Adversarial Supervised Contrastive Learning
• AdvFilter: Predictive Perturbation-aware Filtering against Adversarial Attack via Multi-domain Learning
• Deep Repulsive Prototypes for Adversarial Robustness
• Lower Bounds on Cross-Entropy Loss in the Presence of Test-time Adversaries
• Understanding Robustness in Teacher-Student Setting: A New Perspective
• Semantics-Preserving Adversarial Training
• Training Efficiency and Robustness in Deep Learning
• Less is More: Feature Selection for Adversarial Robustness with Compressive Counter-Adversarial Attacks
• Countering Adversarial Examples: Combining Input Transformation and Noisy Training
• Identifying Layers Susceptible to Adversarial Attacks