Do Wider Neural Networks Really Help Adversarial Robustness?

Abstract

Adversarial training is a powerful type of defense against adversarial examples. Previous empirical results suggest that adversarial training requires wider networks for better performances. However, it remains elusive how neural network width affects model robustness. In this paper, we carefully examine the relationship between network width and model robustness. Specifically, we show that the model robustness is closely related to the tradeoff between natural accuracy and perturbation stability, which is controlled by the robust regularization parameter $\lambda$. With the same $\lambda$, wider networks can achieve better natural accuracy but worse perturbation stability, leading to a potentially worse overall model robustness. To understand the origin of this phenomenon, we further relate the perturbation stability with the network's local Lipschitzness. By leveraging recent results on neural tangent kernels, we theoretically show that wider networks tend to have worse perturbation stability. Our analyses suggest that: 1) the common strategy of first fine-tuning $\lambda$ on small networks and then directly use it for wide model training could lead to deteriorated model robustness; 2) one needs to properly enlarge $\lambda$ to unleash the robustness potential of wider models fully. Finally, we propose a new Width Adjusted Regularization (WAR) method that adaptively enlarges $\lambda$ on wide models and significantly saves the tuning time.

arXiv comment24 pages, 6 tables, 9 figures

References in corpus (31)

• Wide Residual Networks
• ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Theoretically Principled Trade-off between Robustness and Accuracy
• A Convergence Theory for Deep Learning via Over-Parameterization
• Towards Deep Neural Network Architectures Robust to Adversarial Examples
• Certified Adversarial Robustness via Randomized Smoothing
• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• Adversarially Robust Generalization Requires More Data
• Fast is better than free: Revisiting adversarial training
• The Space of Transferable Adversarial Examples
• Countering Adversarial Images using Input Transformations
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models
• Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
• Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers
• Prior Convictions: Black-Box Adversarial Attacks with Bandits and Priors
• Stochastic Activation Pruning for Robust Adversarial Defense
• Adversarial Robustness through Local Linearization
• On the Convergence and Robustness of Adversarial Training
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• Adversarial Training Can Hurt Generalization
• Adversarial examples from computational constraints
• Adversarial Robustness May Be at Odds With Simplicity
• Boosting Adversarial Training with Hypersphere Embedding
• Convergence of Adversarial Training in Overparametrized Neural Networks
• Understanding and Mitigating the Tradeoff Between Robustness and Accuracy
• Adversarial Spheres

Cited by in corpus (9)

• MedViT: A Robust Vision Transformer for Generalized Medical Image Classification
• Towards Robust Vision Transformer
• Shift Invariance Can Reduce Adversarial Robustness
• Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better
• Improving Robustness using Generated Data
• Mind the box: $l_1$-APGD for sparse adversarial attacks on image classifiers
• Improving Model Robustness with Latent Distribution Locally and Globally
• Interpolation can hurt robust generalization even when there is no noise
• Dual Head Adversarial Training