687 citations · 964 across the 9 of their papers we have counts for
21 papers · 1 filter
NormFormer: Improved Transformer Pretraining with Extra Normalization
Sam Shleifer, Jason Weston, Myle Ott
During pretraining, the Pre-LayerNorm transformer suffers from a gradient magnitude mismatch: gradients at early layers are much larger than at later layers. These issues can be al…
Larger-Scale Transformers for Multilingual Masked Language Modeling
Naman Goyal, Jingfei Du, Myle Ott +2
Recent work has demonstrated the effectiveness of cross-lingual language model pretraining for cross-lingual understanding. In this study, we present the results of two larger mult…
Recipes for building an open-domain chatbot
Stephen Roller, Emily Dinan, Naman Goyal +9
Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that scaling neural models in the number of parameters and the size of…
General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference
Jingfei Du, Myle Ott, Haoran Li +2
The state of the art on many NLP tasks is currently achieved by large pre-trained language models, which require a considerable amount of computation. We explore a setting where ma…
Residual Energy-Based Models for Text Generation
Yuntian Deng, Anton Bakhtin, Myle Ott +2
Text generation is ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation. The dominant parametric approach is based on locally normalized models whi…
Residual Energy-Based Models for Text
Anton Bakhtin, Yuntian Deng, Sam Gross +3
Current large-scale auto-regressive language models display impressive fluency and can generate convincing text. In this work we start by asking the question: Can the generations o…