Universal Lipschitz Approximation in Bounded Depth Neural Networks

Abstract

Adversarial attacks against machine learning models are a rather hefty obstacle to our increasing reliance on these models. Due to this, provably robust (certified) machine learning models are a major topic of interest. Lipschitz continuous models present a promising approach to solving this problem. By leveraging the expressive power of a variant of neural networks which maintain low Lipschitz constants, we prove that three layer neural networks using the FullSort activation function are Universal Lipschitz function Approximators (ULAs). This both explains experimental results and paves the way for the creation of better certified models going forward. We conclude by presenting experimental results that suggest that ULAs are a not just a novelty, but a competitive approach to providing certified classifiers, using these results to motivate several potential topics of further research.

References in corpus (12)

• Explaining and Harnessing Adversarial Examples
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Going Deeper with Convolutions
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Certified Defenses against Adversarial Examples
• Scaling provable adversarial defenses
• Sorting out Lipschitz function approximation
• Lipschitz regularized Deep Neural Networks generalize and are adversarially robust
• Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)
• Limitations of the Lipschitz constant as a defense against adversarial examples
• Maximum Resilience of Artificial Neural Networks

Cited by in corpus (7)

• Globally-Robust Neural Networks
• Learning Security Classifiers with Verified Global Robustness Properties
• Interval Universal Approximation for Neural Networks
• Some open questions on morphological operators and representations in the deep learning era
• Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons
• Universal Approximation with Certified Networks
• Universal Approximation of Residual Flows in Maximum Mean Discrepancy