Sorting out Lipschitz function approximation

Abstract

Training neural networks under a strict Lipschitz constraint is useful for provable adversarial robustness, generalization bounds, interpretable gradients, and Wasserstein distance estimation. By the composition property of Lipschitz functions, it suffices to ensure that each individual affine transformation or nonlinear activation is 1-Lipschitz. The challenge is to do this while maintaining the expressive power. We identify a necessary property for such an architecture: each of the layers must preserve the gradient norm during backpropagation. Based on this, we propose to combine a gradient norm preserving activation function, GroupSort, with norm-constrained weight matrices. We show that norm-constrained GroupSort architectures are universal Lipschitz function approximators. Empirically, we show that norm-constrained GroupSort networks achieve tighter estimates of Wasserstein distance than their ReLU counterparts and can achieve provable adversarial robustness guarantees with little cost to accuracy.

arXiv comment8 main pages, 21 pages total, 17 figures. Accepted at ICML 2019

References in corpus (18)

• Spectral Normalization for Generative Adversarial Networks
• Variational Inference with Normalizing Flows
• On the Number of Linear Regions of Deep Neural Networks
• Robustness May Be at Odds with Accuracy
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Robust Large Margin Deep Neural Networks
• On Convergence and Stability of GANs
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
• The Singular Values of Convolutional Layers
• Sylvester Normalizing Flows for Variational Inference
• Full-Capacity Unitary Recurrent Neural Networks
• Learning Generative Models with Sinkhorn Divergences
• Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
• Primal-Dual Wasserstein GAN
• Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)
• Limitations of the Lipschitz constant as a defense against adversarial examples
• Information Geometry of Orthogonal Initializations and Training

Cited by in corpus (26)

• Invertible Residual Networks
• FSPool: Learning Set Representations with Featurewise Sort Pooling
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Parseval Proximal Neural Networks
• Wasserstein Generative Adversarial Uncertainty Quantification in Physics-Informed Neural Networks
• Learning Security Classifiers with Verified Global Robustness Properties
• Lipschitz regularized Deep Neural Networks generalize and are adversarially robust
• Some Theoretical Insights into Wasserstein GANs
• Defending Against Adversarial Examples with K-Nearest Neighbor
• GraN-GAN: Piecewise Gradient Normalization for Generative Adversarial Networks
• Universal Lipschitz Approximation in Bounded Depth Neural Networks
• Robust and Provably Monotonic Networks
• Improved, Deterministic Smoothing for L_1 Certified Robustness
• Some open questions on morphological operators and representations in the deep learning era
• Second-Order Provable Defenses against Adversarial Attacks
• A Closer Look at Double Backpropagation
• Statistical analysis of Wasserstein GANs with applications to time series forecasting
• Markov-Lipschitz Deep Learning
• Towards Expressive Priors for Bayesian Neural Networks: Poisson Process Radial Basis Function Networks
• Lipschitz Bounds and Provably Robust Training by Laplacian Smoothing
• Deep Manifold Transformation for Nonlinear Dimensionality Reduction
• Insta-RS: Instance-wise Randomized Smoothing for Improved Robustness and Accuracy
• Orthogonal Wasserstein GANs
• Lipschitz Constrained GANs via Boundedness and Continuity
• NeuralDP Differentially private neural networks by design
• Regularized Training and Tight Certification for Randomized Smoothed Classifier with Provable Robustness