5.7k citations · 7.1k across the 26 of their papers we have counts for
3 papers · 1 filter
Rethinking embedding coupling in pre-trained language models
Hyung Won Chung, Thibault Févry, Henry Tsai +2
We re-evaluate the standard practice of sharing weights between input and output embeddings in state-of-the-art pre-trained language models. We show that decoupled embeddings provi…
Explicit Alignment Objectives for Multilingual Bidirectional Encoders
Junjie Hu, Melvin Johnson, Orhan Firat +2
Pre-trained cross-lingual encoders such as mBERT (Devlin et al., 2019) and XLMR (Conneau et al., 2020) have proven to be impressively effective at enabling transfer-learning of NLP…
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
Junjie Hu, Sebastian Ruder, Aditya Siddhant +3
Much recent progress in applications of machine learning models to NLP has been driven by benchmarks that evaluate models across a wide variety of tasks. However, these broad-cover…