Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks
arXiv:2110.03825
Abstract
Deep neural networks (DNNs) are known to be vulnerable to adversarial attacks. A range of defense methods have been proposed to train adversarially robust DNNs, among which adversarial training has demonstrated promising results. However, despite preliminary understandings developed for adversarial training, it is still not clear, from the architectural perspective, what configurations can lead to more robust DNNs. In this paper, we address this gap via a comprehensive investigation on the impact of network width and depth on the robustness of adversarially trained DNNs. Specifically, we make the following key observations: 1) more parameters (higher model capacity) does not necessarily help adversarial robustness; 2) reducing capacity at the last stage (the last group of blocks) of the network can actually improve adversarial robustness; and 3) under the same parameter budget, there exists an optimal architectural configuration for adversarial robustness. We also provide a theoretical analysis explaning why such network configuration can help robustness. These architectural insights can help design adversarially robust DNNs. Code is available at \url{https://github.com/HanxunH/RobustWRN}.
arXiv commentNeurIPS 2021
References in corpus (26)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- DARTS: Differentiable Architecture Search
- Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
- Theoretically Principled Trade-off between Robustness and Accuracy
- Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
- Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systems
- Adversarially Robust Generalization Requires More Data
- Fast is better than free: Revisiting adversarial training
- Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality
- On the Convergence and Robustness of Adversarial Training
- Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
- Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
- MMA Training: Direct Input Space Margin Maximization through Adversarial Training
- Adversarially Robust Generalization Just Requires More Unlabeled Data
- Adversarial Robustness May Be at Odds With Simplicity
- Adversarial Robustness as a Prior for Learned Representations
- Improving Adversarial Robustness via Channel-wise Activation Suppressing
- RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
- Adversarially Robust Neural Architectures
- Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
- L2-Nonexpansive Neural Networks
- Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defenses
- Adversarial Camouflage: Hiding Physical-World Attacks with Natural Styles
- DSRNA: Differentiable Search of Robust Neural Architectures
- An Analysis of the Expressiveness of Deep Neural Network Architectures Based on Their Lipschitz Constants
- Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better