Sparse DNNs with Improved Adversarial Robustness

Abstract

Deep neural networks (DNNs) are computationally/memory-intensive and vulnerable to adversarial attacks, making them prohibitive in some real-world applications. By converting dense models into sparse ones, pruning appears to be a promising solution to reducing the computation/memory cost. This paper studies classification models, especially DNN-based ones, to demonstrate that there exists intrinsic relationships between their sparsity and adversarial robustness. Our analyses reveal, both theoretically and empirically, that nonlinear DNN-based classifiers behave differently under $l_2$ attacks from some linear ones. We further demonstrate that an appropriately higher model sparsity implies better robustness of nonlinear DNNs, whereas over-sparsified models can be more difficult to resist adversarial examples.

l1 regularization on weights --> l1 regularization on activations

Cited by in corpus (33)

• What Do Compressed Deep Neural Networks Forget?
• How Can We Be So Dense? The Benefits of Using Highly Sparse Representations
• A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions
• Characterising Bias in Compressed Models
• High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference
• HYDRA: Pruning Adversarially Robust Neural Networks
• Towards Compact and Robust Deep Neural Networks
• Adversarial Neural Pruning with Latent Vulnerability Suppression
• Robust Sparse Regularization: Simultaneously Optimizing Neural Network Robustness and Compactness
• Achieving Adversarial Robustness via Sparsity
• CIFS: Improving Adversarial Robustness of CNNs via Channel-wise Importance-based Feature Selection
• Pruning a restricted Boltzmann machine for quantum state reconstruction
• Gradient ${\ell }_1$ Regularization for Quantization Robustness
• Towards Practical Lottery Ticket Hypothesis for Adversarial Training
• Understanding Adversarial Robustness: The Trade-off between Minimum and Average Margin
• Robustness and Transferability of Universal Attacks on Compressed Models
• On the Effect of Low-Rank Weights on Adversarial Robustness of Neural Networks
• Adversarial Robustness of Supervised Sparse Coding
• Recent Advances in Understanding Adversarial Robustness of Deep Neural Networks
• Adversarially Robust Estimate and Risk Analysis in Linear Regression
• Double Backpropagation for Training Autoencoders against Adversarial Attack
• Brain-inspired reverse adversarial examples
• Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations
• Yet Another Intermediate-Level Attack
• Representation Quality Of Neural Networks Links To Adversarial Attacks and Defences
• Recent Advances in Large Margin Learning
• Spatio-Temporal Sparsification for General Robust Graph Convolution Networks
• The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks
• Dual Head Adversarial Training
• From deep to Shallow: Equivalent Forms of Deep Networks in Reproducing Kernel Krein Space and Indefinite Support Vector Machines
• Deep Minimax Probability Machine
• Guess First to Enable Better Compression and Adversarial Robustness