Batch Normalization is a Cause of Adversarial Vulnerability

Abstract

Batch normalization (batch norm) is often used in an attempt to stabilize and accelerate training in deep neural networks. In many cases it indeed decreases the number of parameter updates required to achieve low training error. However, it also reduces robustness to small adversarial input perturbations and noise by double-digit percentages, as we show on five standard datasets. Furthermore, substituting weight decay for batch norm is sufficient to nullify the relationship between adversarial vulnerability and the input dimension. Our work is consistent with a mean-field analysis that found that batch norm causes exploding gradients.

To appear in the ICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Opening the Black Box of Deep Neural Networks via Information
• Towards the first adversarially robust neural network model on MNIST
• L2 Regularization versus Batch and Weight Normalization
• Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
• advertorch v0.1: An Adversarial Robustness Toolbox based on PyTorch
• A Boundary Tilting Persepective on the Phenomenon of Adversarial Examples
• Adversarial Examples Are a Natural Consequence of Test Error in Noise
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• First-order Adversarial Vulnerability of Neural Networks and Input Dimension
• Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness
• Adversarial Training Versus Weight Decay

Cited by in corpus (29)

• Self-training with Noisy Student improves ImageNet classification
• Improving robustness against common corruptions by covariate shift adaptation
• Smooth Adversarial Training
• Improving Adversarial Robustness via Channel-wise Activation Suppressing
• Uses and Abuses of the Cross-Entropy Loss: Case Studies in Modern Deep Learning
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks
• Dirty Road Can Attack: Security of Deep Learning based Automated Lane Centering under Physical-World Attack
• Quantifying the Preferential Direction of the Model Gradient in Adversarial Training With Projected Gradient Descent
• Adversarial Lipschitz Regularization
• Towards an Adversarially Robust Normalization Approach
• Bit Error Robustness for Energy-Efficient DNN Accelerators
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• Batch Normalization Preconditioning for Neural Network Training
• Manipulating SGD with Data Ordering Attacks
• Shift Invariance Can Reduce Adversarial Robustness
• SOAR: Second-Order Adversarial Regularization
• Dealing with Adversarial Player Strategies in the Neural Network Game iNNk through Ensemble Learning
• No Surprises: Training Robust Lung Nodule Detection for Low-Dose CT Scans by Augmenting with Adversarial Attacks
• Bridging the Gap Between Adversarial Robustness and Optimization Bias
• Batch norm with entropic regularization turns deterministic autoencoders into generative models
• Parameter Prediction for Unseen Deep Architectures
• Scaling Object Detection by Transferring Classification Weights
• An Empirical Study of Batch Normalization and Group Normalization in Conditional Computation
• Clustering Effect of (Linearized) Adversarial Robust Models
• A Useful Taxonomy for Adversarial Robustness of Neural Networks
• New Interpretations of Normalization Methods in Deep Learning
• Training Efficiency and Robustness in Deep Learning
• Farkas layers: don't shift the data, fix the geometry