6 citations · 8 across the 6 of their papers we have counts for
Showing 2023Show all
3 papers · 1 filter
cs.CL2023
Gradient Sparsification For Masked Fine-Tuning of Transformers
James O' Neill, Sourav Dutta
Fine-tuning pretrained self-supervised language models is widely adopted for transfer learning to downstream tasks. Fine-tuning can be achieved by freezing gradients of the pretrai…
cs.CL2023★ 1 cited
Attention over pre-trained Sentence Embeddings for Long Document Classification
Amine Abdaoui, Sourav Dutta
Despite being the current de-facto models in most NLP tasks, transformers are often limited to short sequences due to their quadratic attention complexity on the number of tokens.…
cs.CL2023
Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models
James O' Neill, Sourav Dutta
We investigate the effects of post-training quantization and quantization-aware training on the generalization of Transformer language models. We present a new method called self-d…