Towards Compact and Robust Deep Neural Networks

Abstract

Deep neural networks have achieved impressive performance in many applications but their large number of parameters lead to significant computational and storage overheads. Several recent works attempt to mitigate these overheads by designing compact networks using pruning of connections. However, we observe that most of the existing strategies to design compact networks fail to preserve network robustness against adversarial examples. In this work, we rigorously study the extension of network pruning strategies to preserve both benign accuracy and robustness of a network. Starting with a formal definition of the pruning procedure, including pre-training, weights pruning, and fine-tuning, we propose a new pruning method that can create compact networks while preserving both benign accuracy and robustness. Our method is based on two main insights: (1) we ensure that the training objectives of the pre-training and fine-tuning steps match the training objective of the desired robust model (e.g., adversarial robustness/verifiable robustness), and (2) we keep the pruning strategy agnostic to pre-training and fine-tuning objectives. We evaluate our method on four different networks on the CIFAR-10 dataset and measure benign accuracy, empirical robust accuracy, and verifiable robust accuracy. We demonstrate that our pruning method can preserve on average 93\% benign accuracy, 92.5\% empirical robust accuracy, and 85.0\% verifiable robust accuracy while compressing the tested network by 10$\times$.

14 pages, 9 figures, 7 tables

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• End to End Learning for Self-Driving Cars
• Wide Residual Networks
• Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model
• An Analysis of Deep Neural Network Models for Practical Applications
• Theoretically Principled Trade-off between Robustness and Accuracy
• Provable defenses against adversarial examples via the convex outer adversarial polytope
• On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models
• Semidefinite relaxations for certifying robustness to adversarial examples
• Efficient Formal Safety Analysis of Neural Networks
• Sparse DNNs with Improved Adversarial Robustness
• RANDOM MASK: Towards Robust Convolutional Neural Networks

Cited by in corpus (8)

• What Do Compressed Deep Neural Networks Forget?
• Characterising Bias in Compressed Models
• HYDRA: Pruning Adversarially Robust Neural Networks
• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• Robust Lottery Tickets for Pre-trained Language Models
• Achieving Adversarial Robustness via Sparsity
• Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustness
• Towards Practical Lottery Ticket Hypothesis for Adversarial Training