The Search for Sparse, Robust Neural Networks
arXiv:1912.02386
Abstract
Recent work on deep neural network pruning has shown there exist sparse subnetworks that achieve equal or improved accuracy, training time, and loss using fewer network parameters when compared to their dense counterparts. Orthogonal to pruning literature, deep neural networks are known to be susceptible to adversarial examples, which may pose risks in security- or safety-critical applications. Intuition suggests that there is an inherent trade-off between sparsity and robustness such that these characteristics could not co-exist. We perform an extensive empirical evaluation and analysis testing the Lottery Ticket Hypothesis with adversarial training and show this approach enables us to find sparse, robust neural networks. Code for reproducing experiments is available here: https://github.com/justincosentino/robust-sparse-networks.
The Safety and Robustness in Decision Making Workshop at the 33rd Conference on Neural InformationProcessing Systems (NeurIPS 2019), Vancouver, Canada
References in corpus (10)
- Adam: A Method for Stochastic Optimization
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Towards Deep Learning Models Resistant to Adversarial Attacks
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- Pruning Filters for Efficient ConvNets
- Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
- Robustness May Be at Odds with Accuracy
- Stabilizing the Lottery Ticket Hypothesis
- Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability
Cited by in corpus (6)
- Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient
- Achieving Adversarial Robustness via Sparsity
- Pruning via Iterative Ranking of Sensitivity Statistics
- Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustness
- GANs Can Play Lottery Tickets Too
- Spatio-Temporal Sparsification for General Robust Graph Convolution Networks