Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks
arXiv:1906.04893
Abstract
Tight estimation of the Lipschitz constant for deep neural networks (DNNs) is useful in many applications ranging from robustness certification of classifiers to stability analysis of closed-loop systems with reinforcement learning controllers. Existing methods in the literature for estimating the Lipschitz constant suffer from either lack of accuracy or poor scalability. In this paper, we present a convex optimization framework to compute guaranteed upper bounds on the Lipschitz constant of DNNs both accurately and efficiently. Our main idea is to interpret activation functions as gradients of convex potential functions. Hence, they satisfy certain properties that can be described by quadratic constraints. This particular description allows us to pose the Lipschitz constant estimation problem as a semidefinite program (SDP). The resulting SDP can be adapted to increase either the estimation accuracy (by capturing the interaction between activation functions of different layers) or scalability (by decomposition and parallel implementation). We illustrate the utility of our approach with a variety of experiments on randomly generated networks and on classifiers trained on the MNIST and Iris datasets. In particular, we experimentally demonstrate that our Lipschitz bounds are the most accurate compared to those in the literature. We also study the impact of adversarial training methods on the Lipschitz bounds of the resulting classifiers and show that our bounds can be used to efficiently provide robustness guarantees.
References in corpus (16)
- Explaining and Harnessing Adversarial Examples
- Towards Deep Learning Models Resistant to Adversarial Attacks
- Certified Defenses against Adversarial Examples
- Efficient Neural Network Robustness Certification with General Activation Functions
- Evaluating Robustness of Neural Networks with Mixed Integer Programming
- Safe Model-based Reinforcement Learning with Stability Guarantees
- Semidefinite relaxations for certifying robustness to adversarial examples
- Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach
- Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
- Towards Fast Computation of Certified Robustness for ReLU Networks
- Reachability Analysis of Deep Neural Networks with Provable Guarantees
- L2-Nonexpansive Neural Networks
- Stability-certified reinforcement learning: A control-theoretic perspective
- Provable Certificates for Adversarial Examples: Fitting a Ball in the Union of Polytopes
- Lipschitz Properties for Deep Convolutional Networks
- On Lipschitz Bounds of General Convolutional Neural Networks
Cited by in corpus (9)
- Globally-Robust Neural Networks
- Adversarial Learning in Statistical Classification: A Comprehensive Review of Defenses Against Attacks
- There is Limited Correlation between Coverage and Robustness for Deep Neural Networks
- Verification of Neural Network Control Policy Under Persistent Adversarial Perturbation
- Over-the-Air Federated Learning with Retransmissions (Extended Version)
- Sensitivity analysis in differentially private machine learning using hybrid automatic differentiation
- On the Robustness and Generalization of Deep Learning Driven Full Waveform Inversion
- Bridged Adversarial Training
- Notes on Margin Training and Margin p-Values for Deep Neural Network Classifiers