602 citations · 1.4k across the 21 of their papers we have counts for
4 papers · 1 filter
DOCmT5: Document-Level Pretraining of Multilingual Language Models
Chia-Hsuan Lee, Aditya Siddhant, Viresh Ratnakar +1
In this paper, we introduce DOCmT5, a multilingual sequence-to-sequence language model pretrained with large scale parallel documents. While previous approaches have focused on lev…
nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?
Mihir Kale, Aditya Siddhant, Noah Constant +3
Recently, mT5 - a massively multilingual version of T5 - leveraged a unified text-to-text format to attain state-of-the-art results on a wide variety of multilingual NLP tasks. In…
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
Sebastian Ruder, Noah Constant, Jan Botha +8
Machine learning has brought striking advances in multilingual natural language processing capabilities over the past year. For example, the latest techniques have improved the sta…
Distilling Large Language Models into Tiny and Effective Students using pQRNN
Prabhu Kaliamoorthi, Aditya Siddhant, Edward Li +1
Large pre-trained multilingual models like mBERT, XLM-R achieve state of the art results on language understanding tasks. However, they are not well suited for latency critical app…