Adversarial Robustness Guarantees for Random Deep Neural Networks
arXiv:2004.05923
Abstract
The reliability of deep learning algorithms is fundamentally challenged by the existence of adversarial examples, which are incorrectly classified inputs that are extremely close to a correctly classified input. We explore the properties of adversarial examples for deep neural networks with random weights and biases, and prove that for any , the distance of any given input from the classification boundary scales as one over the square root of the dimension of the input times the norm of the input. The results are based on the recently proved equivalence between Gaussian processes and deep neural networks in the limit of infinite width of the hidden layers, and are validated with experiments on both random deep neural networks and deep neural networks trained on the MNIST and CIFAR10 datasets. The results constitute a fundamental advance in the theoretical understanding of adversarial examples, and open the way to a thorough theoretical characterization of the relation between network architecture and robustness to adversarial perturbations.
References in corpus (37)
- Adam: A Method for Stochastic Optimization
- Deep Learning in Neural Networks: An Overview
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Delving into Transferable Adversarial Examples and Black-box Attacks
- The Loss Surfaces of Multilayer Networks
- Provable defenses against adversarial examples via the convex outer adversarial polytope
- Certified Adversarial Robustness via Randomized Smoothing
- Certified Defenses against Adversarial Examples
- Deep Neural Networks as Gaussian Processes
- Adversarially Robust Generalization Requires More Data
- Robust Physical-World Attacks on Deep Learning Models
- On Exact Computation with an Infinitely Wide Neural Net
- Exponential expressivity in deep neural networks through transient chaos
- Towards the first adversarially robust neural network model on MNIST
- Certified Adversarial Robustness with Additive Noise
- Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Adversarial vulnerability for any classifier
- Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
- A Boundary Tilting Persepective on the Phenomenon of Adversarial Examples
- Adversarial Examples Are a Natural Consequence of Test Error in Noise
- Adversarial Training Can Hurt Generalization
- Adversarial examples from computational constraints
- Robustness of classifiers: from adversarial to random noise
- Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability
- The Emergence of Spectral Universality in Deep Networks
- Enhanced Convolutional Neural Tangent Kernels
- Adversarial Robustness May Be at Odds With Simplicity
- A Simple Explanation for the Existence of Adversarial Examples with Small Hamming Distance
- Convergence of Adversarial Training in Overparametrized Neural Networks
- Understanding Priors in Bayesian Neural Networks at the Unit Level
- Random deep neural networks are biased towards simple functions
- Bridging Adversarial Robustness and Gradient Interpretability
- Provable Certificates for Adversarial Examples: Fitting a Ball in the Union of Polytopes
- An ADMM-Based Universal Framework for Adversarial Attacks on Deep Neural Networks
- Bayesian Adversarial Spheres: Bayesian Inference and Adversarial Examples in a Noiseless Setting