Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons

Abstract

It is well-known that standard neural networks, even with a high classification accuracy, are vulnerable to small ${\ell }_{\infty }$-norm bounded adversarial perturbations. Although many attempts have been made, most previous works either can only provide empirical verification of the defense to a particular attack method, or can only develop a certified guarantee of the model robustness in limited scenarios. In this paper, we seek for a new approach to develop a theoretically principled neural network that inherently resists ${\ell }_{\infty }$ perturbations. In particular, we design a novel neuron that uses ${\ell }_{\infty }$-distance as its basic operation (which we call ${\ell }_{\infty }$-dist neuron), and show that any neural network constructed with ${\ell }_{\infty }$-dist neurons (called ${\ell }_{\infty }$-dist net) is naturally a 1-Lipschitz function with respect to ${\ell }_{\infty }$-norm. This directly provides a rigorous guarantee of the certified robustness based on the margin of prediction outputs. We then prove that such networks have enough expressive power to approximate any 1-Lipschitz function with robust generalization guarantee. We further provide a holistic training strategy that can greatly alleviate optimization difficulties. Experimental results show that using ${\ell }_{\infty }$-dist nets as basic building blocks, we consistently achieve state-of-the-art performance on commonly used datasets: 93.09% certified accuracy on MNIST ($\epsilon =0.3$), 35.42% on CIFAR-10 ($\epsilon =8/255$) and 16.31% on TinyImageNet ($\epsilon =1/255$).

arXiv commentAppearing at International Conference on Machine Learning (ICML) 2021

References in corpus (29)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Theoretically Principled Trade-off between Robustness and Accuracy
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Certified Defenses against Adversarial Examples
• Fast is better than free: Revisiting adversarial training
• Efficient Neural Network Robustness Certification with General Activation Functions
• Evaluating Robustness of Neural Networks with Mixed Integer Programming
• Exploring Generalization in Deep Learning
• On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models
• Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers
• Learning with a Strong Adversary
• Robust Large Margin Deep Neural Networks
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks
• Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
• On Adaptive Attacks to Adversarial Example Defenses
• Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability
• MMA Training: Direct Input Space Margin Maximization through Adversarial Training
• Training verified learners with learned verifiers
• Curse of Dimensionality on Randomized Smoothing for Certifiable Robustness
• Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks
• Randomized Smoothing of All Shapes and Sizes
• Kernel Based Progressive Distillation for Adder Neural Networks
• Black-Box Certification with Randomized Smoothing: A Functional Optimization Based Framework
• Random Smoothing Might be Unable to Certify ${\ell }_{\infty }$ Robustness for High-Dimensional Images
• A Provable Defense for Deep Residual Networks
• Universal Lipschitz Approximation in Bounded Depth Neural Networks
• Limitations of the Lipschitz constant as a defense against adversarial examples