Blind Adversarial Pruning: Balance Accuracy, Efficiency and Robustness
arXiv:2004.05913 · doi:10.1109/ICME51207.2021.9428149
Abstract
With the growth of interest in the attack and defense of deep neural networks, researchers are focusing more on the robustness of applying them to devices with limited memory. Thus, unlike adversarial training, which only considers the balance between accuracy and robustness, we come to a more meaningful and critical issue, i.e., the balance among accuracy, efficiency and robustness (AER). Recently, some related works focused on this issue, but with different observations, and the relations among AER remain unclear. This paper first investigates the robustness of pruned models with different compression ratios under the gradual pruning process and concludes that the robustness of the pruned model drastically varies with different pruning processes, especially in response to attacks with large strength. Second, we test the performance of mixing the clean data and adversarial examples (generated with a prescribed uniform budget) into the gradual pruning process, called adversarial pruning, and find the following: the pruned model's robustness exhibits high sensitivity to the budget. Furthermore, to better balance the AER, we propose an approach called blind adversarial pruning (BAP), which introduces the idea of blind adversarial training into the gradual pruning process. The main idea is to use a cutoff-scale strategy to adaptively estimate a nonuniform budget to modify the AEs used during pruning, thus ensuring that the strengths of AEs are dynamically located within a reasonable range at each pruning step and ultimately improving the overall AER of the pruned model. The experimental results obtained using BAP for pruning classification models based on several benchmarks demonstrate the competitive performance of this method: the robustness of the model pruned by BAP is more stable among varying pruning processes, and BAP exhibits better overall AER than adversarial pruning.
References in corpus (23)
- Adam: A Method for Stochastic Optimization
- Distilling the Knowledge in a Neural Network
- Sequence to Sequence Learning with Neural Networks
- Explaining and Harnessing Adversarial Examples
- Intriguing properties of neural networks
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Ensemble Adversarial Training: Attacks and Defenses
- FitNets: Hints for Thin Deep Nets
- BinaryConnect: Training Deep Neural Networks with binary weights during propagations
- Do Deep Nets Really Need to be Deep?
- Theoretically Principled Trade-off between Robustness and Accuracy
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Compressing Neural Networks with the Hashing Trick
- Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
- Convolutional neural networks with low-rank regularization
- Adversarial Examples: Opportunities and Challenges
- Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
- Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability
- MMA Training: Direct Input Space Margin Maximization through Adversarial Training
- Improving the Generalization of Adversarial Training with Domain Adaptation
- Unrestricted Adversarial Examples
- Adversarial Neural Pruning with Latent Vulnerability Suppression