RecurJac: An Efficient Recursive Algorithm for Bounding Jacobian Matrix of Neural Networks and Its Applications
arXiv:1810.11783
Abstract
The Jacobian matrix (or the gradient for single-output networks) is directly related to many important properties of neural networks, such as the function landscape, stationary points, (local) Lipschitz constants and robustness to adversarial attacks. In this paper, we propose a recursive algorithm, RecurJac, to compute both upper and lower bounds for each element in the Jacobian matrix of a neural network with respect to network's input, and the network can contain a wide range of activation functions. As a byproduct, we can efficiently obtain a (local) Lipschitz constant, which plays a crucial role in neural network robustness verification, as well as the training stability of GANs. Experiments show that (local) Lipschitz constants produced by our method is of better quality than previous approaches, thus providing better robustness verification results. Our algorithm has polynomial time complexity, and its computation time is reasonable even for relatively large networks. Additionally, we use our bounds of Jacobian matrix to characterize the landscape of the neural network, for example, to determine whether there exist stationary points in a local neighborhood. Source code available at \url{http://github.com/huanzhang12/RecurJac-Jacobian-bounds}.
arXiv commentWork done during internship at Microsoft Research
References in corpus (11)
- Spectral Normalization for Generative Adversarial Networks
- Provable defenses against adversarial examples via the convex outer adversarial polytope
- Certified Defenses against Adversarial Examples
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
- Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach
- Do GANs actually learn the distribution? An empirical study
- On integral probability metrics, ϕ-divergences and binary classification
- On the Discrimination-Generalization Tradeoff in GANs
- L2-Nonexpansive Neural Networks
- Large Margin Deep Networks for Classification