Gradient Sparsification For Masked Fine-Tuning of Transformers

James O' Neill, Sourav Dutta

Fine-tuning pretrained self-supervised language models is widely adopted for transfer learning to downstream tasks. Fine-tuning can be achieved by freezing gradients of the pretrai…