Towards Practical Lottery Ticket Hypothesis for Adversarial Training
arXiv:2003.05733
Abstract
Recent research has proposed the lottery ticket hypothesis, suggesting that for a deep neural network, there exist trainable sub-networks performing equally or better than the original model with commensurate training steps. While this discovery is insightful, finding proper sub-networks requires iterative training and pruning. The high cost incurred limits the applications of the lottery ticket hypothesis. We show there exists a subset of the aforementioned sub-networks that converge significantly faster during the training process and thus can mitigate the cost issue. We conduct extensive experiments to show such sub-networks consistently exist across various model structures for a restrictive setting of hyperparameters (, carefully selected learning rate, pruning ratio, and model capacity). As a practical application of our findings, we demonstrate that such sub-networks can help in cutting down the total time of adversarial training, a standard approach to improve robustness, by up to 49\% on CIFAR-10 to achieve the state-of-the-art robustness.
References in corpus (13)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- Wide Residual Networks
- Do Deep Nets Really Need to be Deep?
- Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples
- Rethinking the Value of Network Pruning
- Delving into Transferable Adversarial Examples and Black-box Attacks
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- Stochastic Activation Pruning for Robust Adversarial Defense
- Stabilizing the Lottery Ticket Hypothesis
- Sparse DNNs with Improved Adversarial Robustness
- The Difficulty of Training Sparse Neural Networks
- Towards Compact and Robust Deep Neural Networks