6 citations · 8 across the 6 of their papers we have counts for
6 papers
Gradient Sparsification For Masked Fine-Tuning of Transformers
James O' Neill, Sourav Dutta
Fine-tuning pretrained self-supervised language models is widely adopted for transfer learning to downstream tasks. Fine-tuning can be achieved by freezing gradients of the pretrai…
Attention over pre-trained Sentence Embeddings for Long Document Classification
Amine Abdaoui, Sourav Dutta
Despite being the current de-facto models in most NLP tasks, transformers are often limited to short sequences due to their quadratic attention complexity on the number of tokens.…
Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models
James O' Neill, Sourav Dutta
We investigate the effects of post-training quantization and quantization-aware training on the generalization of Transformer language models. We present a new method called self-d…
EdinSaar@WMT21: North-Germanic Low-Resource Multilingual NMT
Svetlana Tchistiakova, Jesujoba Alabi, Koel Dutta Chowdhury +2
We describe the EdinSaar submission to the shared task of Multilingual Low-Resource Translation for North Germanic Languages at the Sixth Conference on Machine Translation (WMT2021…
Whatcha lookin' at? DeepLIFTing BERT's Attention in Question Answering
Ekaterina Arkhangelskaia, Sourav Dutta
There has been great success recently in tackling challenging NLP tasks by neural networks which have been pre-trained and fine-tuned on large amounts of task data. In this paper,…
Mapping Supervised Bilingual Word Embeddings from English to low-resource languages
Sourav Dutta
It is very challenging to work with low-resource languages due to the inadequate availability of data. Using a dictionary to map independently trained word embeddings into a shared…