Robust Lottery Tickets for Pre-trained Language Models
arXiv:2211.03013 · doi:10.18653/v1/2022.acl-long.157
Abstract
Recent works on Lottery Ticket Hypothesis have shown that pre-trained language models (PLMs) contain smaller matching subnetworks(winning tickets) which are capable of reaching accuracy comparable to the original models. However, these tickets are proved to be notrobust to adversarial examples, and even worse than their PLM counterparts. To address this problem, we propose a novel method based on learning binary weight masks to identify robust tickets hidden in the original PLMs. Since the loss is not differentiable for the binary mask, we assign the hard concrete distribution to the masks and encourage their sparsity using a smoothing approximation of L0 regularization.Furthermore, we design an adversarial loss objective to guide the search for robust tickets and ensure that the tickets perform well bothin accuracy and robustness. Experimental results show the significant improvement of the proposed method over previous work on adversarial robustness evaluation.
arXiv commentACL 2022. https://aclanthology.org/2022.acl-long.157
References in corpus (14)
- Decoupled Weight Decay Regularization
- Categorical Reparameterization with Gumbel-Softmax
- Character-level Convolutional Networks for Text Classification
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- Comparing Rewinding and Fine-tuning in Neural Network Pruning
- Rigging the Lottery: Making All Tickets Winners
- Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
- InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective
- Towards Compact and Robust Deep Neural Networks
- TAVAT: Token-Aware Virtual Adversarial Training for Language Understanding
- Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
- Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding