L2-Nonexpansive Neural Networks

Abstract

This paper proposes a class of well-conditioned neural networks in which a unit amount of change in the inputs causes at most a unit amount of change in the outputs or any of the internal layers. We develop the known methodology of controlling Lipschitz constants to realize its full potential in maximizing robustness, with a new regularization scheme for linear layers, new ways to adapt nonlinearities and a new loss function. With MNIST and CIFAR-10 classifiers, we demonstrate a number of advantages. Without needing any adversarial training, the proposed classifiers exceed the state of the art in robustness against white-box L2-bounded adversarial attacks. They generalize better than ordinary networks from noisy data with partially random labels. Their outputs are quantitatively meaningful and indicate levels of confidence and generalization, among other desirable properties.

References in corpus (12)

• On the difficulty of training Recurrent Neural Networks
• Ensemble Adversarial Training: Attacks and Defenses
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Certified Defenses against Adversarial Examples
• Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
• Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Did you hear that? Adversarial Examples Against Automatic Speech Recognition
• Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey
• Provably Minimally-Distorted Adversarial Examples
• HotFlip: White-Box Adversarial Examples for Text Classification
• Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning

Cited by in corpus (14)

• On Evaluating Adversarial Robustness
• A Closer Look at Accuracy vs. Robustness
• Testing Robustness Against Unforeseen Adversaries
• You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
• Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks
• Adversarially Robust Neural Architectures
• Defensive Quantization: When Efficiency Meets Robustness
• Defend Deep Neural Networks Against Adversarial Examples via Fixed and Dynamic Quantized Activation Functions
• Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks
• Defending Against Adversarial Examples with K-Nearest Neighbor
• RecurJac: An Efficient Recursive Algorithm for Bounding Jacobian Matrix of Neural Networks and Its Applications
• Certified Defense via Latent Space Randomized Smoothing with Orthogonal Encoders
• Neural Belief Reasoner
• Leveraging Local Variation in Data: Sampling and Weighting Schemes for Supervised Deep Learning