Batch Normalization is a Cause of Adversarial Vulnerability
arXiv:1905.02161
Abstract
Batch normalization (batch norm) is often used in an attempt to stabilize and accelerate training in deep neural networks. In many cases it indeed decreases the number of parameter updates required to achieve low training error. However, it also reduces robustness to small adversarial input perturbations and noise by double-digit percentages, as we show on five standard datasets. Furthermore, substituting weight decay for batch norm is sufficient to nullify the relationship between adversarial vulnerability and the input dimension. Our work is consistent with a mean-field analysis that found that batch norm causes exploding gradients.
To appear in the ICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena
References in corpus (7)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- L2 Regularization versus Batch and Weight Normalization
- Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
- advertorch v0.1: An Adversarial Robustness Toolbox based on PyTorch
- A Boundary Tilting Persepective on the Phenomenon of Adversarial Examples
- Adversarial Examples Are a Natural Consequence of Test Error in Noise
- Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness
Cited by in corpus (5)
- Normalization Techniques in Training DNNs: Methodology, Analysis and Application
- Towards an Adversarially Robust Normalization Approach
- A Useful Taxonomy for Adversarial Robustness of Neural Networks
- Farkas layers: don't shift the data, fix the geometry
- New Interpretations of Normalization Methods in Deep Learning