470 citations · 1.7k across the 40 of their papers we have counts for
9 papers · 1 filter
A Length-Extrapolatable Transformer
Yutao Sun, Li Dong, Barun Patra +6
Position modeling plays a critical role in Transformers. In this paper, we focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. We define…
TorchScale: Transformers at Scale
Shuming Ma, Hongyu Wang, Shaohan Huang +8
Large Transformers have achieved state-of-the-art performance across many tasks. Most open-source libraries on scaling Transformers focus on improving training or inference with be…
Holistic Evaluation of Language Models
Percy Liang, Rishi Bommasani, Tony Lee +47
Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Hol…
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
Barun Patra, Saksham Singhal, Shaohan Huang +5
In this paper, we elaborate upon recipes for building multilingual representation models that are not only competitive with existing state-of-the-art models but are also more param…
Foundation Transformers
Hongyu Wang, Shuming Ma, Shaohan Huang +12
A big convergence of model architectures across language, vision, speech, and multimodal is emerging. However, under the same name "Transformers", the above areas use different imp…
Language Model Decoding as Likelihood-Utility Alignment
Martin Josifoski, Maxime Peyrard, Frano Rajic +8
A critical component of a successful language generation pipeline is the decoding algorithm. However, the general principles that should guide the choice of a decoding algorithm re…
How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?
Shiyue Zhang, Vishrav Chaudhary, Naman Goyal +4
A multilingual tokenizer is a fundamental component of multilingual neural machine translation. It is trained from a multilingual corpus. Since a skewed data distribution is consid…
Data Selection Curriculum for Neural Machine Translation
Tasnim Mohiuddin, Philipp Koehn, Vishrav Chaudhary +3
Neural Machine Translation (NMT) models are typically trained on heterogeneous data that are concatenated and randomly shuffled. However, not all of the training data are equally u…
OCR Improves Machine Translation for Low-Resource Languages
Oana Ignat, Jean Maillard, Vishrav Chaudhary +1
We aim to investigate the performance of current OCR systems on low resource languages and low resource scripts. We introduce and make publicly available a novel benchmark, OCR4MT,…