HYDRA: Pruning Adversarially Robust Neural Networks

Abstract

In safety-critical but computationally resource-constrained applications, deep learning faces two key challenges: lack of robustness against adversarial attacks and large neural network size (often millions of parameters). While the research community has extensively explored the use of robust training and network pruning independently to address one of these challenges, only a few recent works have studied them jointly. However, these works inherit a heuristic pruning strategy that was developed for benign training, which performs poorly when integrated with robust training techniques, including adversarial training and verifiable robust training. To overcome this challenge, we propose to make pruning techniques aware of the robust training objective and let the training objective guide the search for which connections to prune. We realize this insight by formulating the pruning objective as an empirical risk minimization problem which is solved efficiently using SGD. We demonstrate that our approach, titled HYDRA, achieves compressed networks with state-of-the-art benign and robust accuracy, simultaneously. We demonstrate the success of our approach across CIFAR-10, SVHN, and ImageNet dataset with four robust training techniques: iterative adversarial training, randomized smoothing, MixTrain, and CROWN-IBP. We also demonstrate the existence of highly robust sub-networks within non-robust networks. Our code and compressed networks are publicly available at \url{https://github.com/inspire-group/compactness-robustness}.

arXiv commentNeurIPS 2020

References in corpus (22)

• Rethinking the Value of Network Pruning
• Theoretically Principled Trade-off between Robustness and Accuracy
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• Certified Adversarial Robustness via Randomized Smoothing
• Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• Using Pre-Training Can Improve Model Robustness and Uncertainty
• Efficient Neural Network Robustness Certification with General Activation Functions
• Formal Security Analysis of Neural Networks using Symbolic Intervals
• On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models
• Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers
• Certified Adversarial Robustness with Additive Noise
• Efficient Formal Safety Analysis of Neural Networks
• Deep Rewiring: Training very sparse deep networks
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Sparse DNNs with Improved Adversarial Robustness
• What Do Compressed Deep Neural Networks Forget?
• Model Compression with Adversarial Robustness: A Unified Optimization Framework
• Towards Compact and Robust Deep Neural Networks
• Robust Sparse Regularization: Simultaneously Optimizing Neural Network Robustness and Compactness
• Intriguing properties of adversarial training at scale

Cited by in corpus (6)

• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• What You See is Not What the Network Infers: Detecting Adversarial Examples Based on Semantic Contradiction
• DASS: Differentiable Architecture Search for Sparse neural networks
• Pruning via Iterative Ranking of Sensitivity Statistics
• SwitchX: Gmin-Gmax Switching for Energy-Efficient and Robust Implementation of Binary Neural Networks on ReRAM Xbars
• RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation