Towards Stable and Efficient Training of Verifiably Robust Neural Networks

Abstract

Training neural networks with verifiable robustness guarantees is challenging. Several existing approaches utilize linear relaxation based neural network output bounds under perturbation, but they can slow down training by a factor of hundreds depending on the underlying network architectures. Meanwhile, interval bound propagation (IBP) based training is efficient and significantly outperforms linear relaxation based methods on many tasks, yet it may suffer from stability issues since the bounds are much looser especially at the beginning of training. In this paper, we propose a new certified adversarial training method, CROWN-IBP, by combining the fast IBP bounds in a forward bounding pass and a tight linear relaxation based bound, CROWN, in a backward bounding pass. CROWN-IBP is computationally efficient and consistently outperforms IBP baselines on training verifiably robust neural networks. We conduct large scale experiments on MNIST and CIFAR datasets, and outperform all previous linear relaxation and bound propagation based certified defenses in ${\ell }_{\infty }$ robustness. Notably, we achieve 7.02% verified test error on MNIST at $\epsilon =0.3$, and 66.94% on CIFAR-10 with $\epsilon =8/255$. Code is available at https://github.com/deepmind/interval-bound-propagation (TensorFlow) and https://github.com/huanzhang12/CROWN-IBP (PyTorch).

References in corpus (23)

• Theoretically Principled Trade-off between Robustness and Accuracy
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Certified Adversarial Robustness via Randomized Smoothing
• Certified Defenses against Adversarial Examples
• Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality
• Efficient Neural Network Robustness Certification with General Activation Functions
• PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples
• On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models
• Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation
• Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers
• Spatially Transformed Adversarial Examples
• Generating Adversarial Examples with Adversarial Networks
• Semidefinite relaxations for certifying robustness to adversarial examples
• Efficient Formal Safety Analysis of Neural Networks
• A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks
• Structured Adversarial Attack: Towards General Implementation and Better Interpretability
• Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability
• Training verified learners with learned verifiers
• The Limitations of Adversarial Training and the Blind-Spot Attack
• Characterizing Adversarial Examples Based on Spatial Consistency Information for Semantic Segmentation
• A Provable Defense for Deep Residual Networks
• Verification of deep probabilistic models
• Verification of Non-Linear Specifications for Neural Networks

Cited by in corpus (4)

• Learning perturbation sets for robust machine learning
• On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them
• NLP Verification: Towards a General Methodology for Certifying Robustness
• Multi-head Ensemble of Smoothed Classifiers for Certified Robustness