Robust Sparse Regularization: Simultaneously Optimizing Neural Network Robustness and Compactness

Abstract

Deep Neural Network (DNN) trained by the gradient descent method is known to be vulnerable to maliciously perturbed adversarial input, aka. adversarial attack. As one of the countermeasures against adversarial attack, increasing the model capacity for DNN robustness enhancement was discussed and reported as an effective approach by many recent works. In this work, we show that shrinking the model size through proper weight pruning can even be helpful to improve the DNN robustness under adversarial attack. For obtaining a simultaneously robust and compact DNN model, we propose a multi-objective training method called Robust Sparse Regularization (RSR), through the fusion of various regularization techniques, including channel-wise noise injection, lasso weight penalty, and adversarial training. We conduct extensive experiments across popular ResNet-20, ResNet-18 and VGG-16 DNN architectures to demonstrate the effectiveness of RSR against popular white-box (i.e., PGD and FGSM) and black-box attacks. Thanks to RSR, 85% weight connections of ResNet-18 can be pruned while still achieving 0.68% and 8.72% improvement in clean- and perturbed-data accuracy respectively on CIFAR-10 dataset, in comparison to its PGD adversarial training baseline.

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Variational Dropout Sparsifies Deep Neural Networks
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Sparse DNNs with Improved Adversarial Robustness
• Towards Robust Neural Networks via Random Self-ensemble
• Defend Deep Neural Networks Against Adversarial Examples via Fixed and Dynamic Quantized Activation Functions
• Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning
• Structured Pruning for Efficient ConvNets via Incremental Regularization

Cited by in corpus (5)

• HYDRA: Pruning Adversarially Robust Neural Networks
• Achieving Adversarial Robustness via Sparsity
• Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustness
• A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNs
• Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks