Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks

Abstract

High sensitivity of neural networks against malicious perturbations on inputs causes security concerns. To take a steady step towards robust classifiers, we aim to create neural network models provably defended from perturbations. Prior certification work requires strong assumptions on network structures and massive computational costs, and thus the range of their applications was limited. From the relationship between the Lipschitz constants and prediction margins, we present a computationally efficient calculation technique to lower-bound the size of adversarial perturbations that can deceive networks, and that is widely applicable to various complicated networks. Moreover, we propose an efficient training procedure that robustifies networks and significantly improves the provably guarded areas around data points. In experimental evaluations, our method showed its ability to provide a non-trivial guarantee and enhance robustness for even large networks.

To appear in NIPS2018

Cited by in corpus (52)

• Invertible Residual Networks
• Motivating the Rules of the Game for Adversarial Example Research
• Training robust neural networks using Lipschitz bounds
• Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
• MMA Training: Direct Input Space Margin Maximization through Adversarial Training
• Attacks Which Do Not Kill Training Make Adversarial Learning Stronger
• Convolutional Proximal Neural Networks and Plug-and-Play Algorithms
• Parseval Proximal Neural Networks
• Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks
• Adversarial Examples - A Complete Characterisation of the Phenomenon
• Indirect Adversarial Attacks via Poisoning Neighbors for Graph Convolutional Networks
• Geometry-aware Instance-reweighted Adversarial Training
• Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks
• Globally-Robust Neural Networks
• The Lipschitz Constant of Self-Attention
• Why Spectral Normalization Stabilizes GANs: Analysis and Improvements
• Lipschitz regularized Deep Neural Networks generalize and are adversarially robust
• Adversarial Learning in Statistical Classification: A Comprehensive Review of Defenses Against Attacks
• Evading classifiers in discrete domains with provable optimality guarantees
• Provable Certificates for Adversarial Examples: Fitting a Ball in the Union of Polytopes
• Defending Against Adversarial Examples with K-Nearest Neighbor
• Semialgebraic Optimization for Lipschitz Constants of ReLU Networks
• Adversarial Training Versus Weight Decay
• Robust learning with implicit residual networks
• A case for new neural network smoothness constraints
• Calibrated Surrogate Losses for Adversarially Robust Classification
• Fine-grained Synthesis of Unrestricted Adversarial Examples
• Improving Transformation Invariance in Contrastive Representation Learning
• Designing Stable Neural Networks using Convex Analysis and ODEs
• Fantastic Four: Differentiable Bounds on Singular Values of Convolution Layers
• Boosting the Certified Robustness of L-infinity Distance Nets
• Lipschitz Networks and Distributional Robustness
• MetaMixUp: Learning Adaptive Interpolation Policy of MixUp with Meta-Learning
• A Game Theoretic Analysis of Additive Adversarial Attacks and Defenses
• Weight-Covariance Alignment for Adversarially Robust Neural Networks
• Learning Diverse-Structured Networks for Adversarial Robustness
• A Frobenius norm regularization method for convolutional kernels to avoid unstable gradient problem
• Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons
• A PAC-Bayes Analysis of Adversarial Robustness
• Certifiably Robust Variational Autoencoders
• Insta-RS: Instance-wise Randomized Smoothing for Improved Robustness and Accuracy
• nn-dependability-kit: Engineering Neural Networks for Safety-Critical Autonomous Driving Systems
• Smoothness Analysis of Adversarial Training
• Lipschitz Constrained GANs via Boundedness and Continuity
• Continuous Safety Verification of Neural Networks
• On the Structural Sensitivity of Deep Convolutional Networks to the Directions of Fourier Basis Functions
• Safety-Aware Hardening of 3D Object Detection Neural Network Systems
• Variational Autoencoders: A Harmonic Perspective
• Notes on Margin Training and Margin p-Values for Deep Neural Network Classifiers
• Absum: Simple Regularization Method for Reducing Structural Sensitivity of Convolutional Neural Networks
• Recent Advances in Large Margin Learning
• Is the Rush to Machine Learning Jeopardizing Safety? Results of a Survey