933 citations · 6k across the 51 of their papers we have counts for
23 papers · 1 filter
Context Training with Active Information Seeking
Zeyu Huang, Adhiguna Kuncoro, Qixuan Feng +4
Most existing large language models (LLMs) are expensive to adapt after deployment, especially when a task requires newly produced information or niche domain knowledge. Recent wor…
Decoupled DiLoCo for Resilient Distributed Pre-training
Arthur Douillard, Keith Rush, Yani Donchev +14
Modern large-scale language model pre-training relies heavily on the single program multiple data (SPMD) paradigm, which requires tight coupling across accelerators. Due to this co…
Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch
Arthur Douillard, Yanislav Donchev, Keith Rush +11
Training of large language models (LLMs) is typically distributed across a large number of accelerators to reduce training time. Since internal states and parameter gradients need…
Towards Robust and Efficient Continual Language Learning
Adam Fisch, Amal Rannen-Triki, Razvan Pascanu +4
As the application space of language models continues to evolve, a natural question to ask is how we can quickly adapt models to new tasks. We approach this classic question from a…
The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation
Naman Goyal, Cynthia Gao, Vishrav Chaudhary +7
One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks. Current evaluation benchmarks eithe…
Multi-scale Transformer Language Models
Sandeep Subramanian, Ronan Collobert, Marc'Aurelio Ranzato +1
We investigate multi-scale transformer language models that learn representations of text at multiple scales, and present three different architectures that have an inductive bias…
Residual Energy-Based Models for Text
Anton Bakhtin, Yuntian Deng, Sam Gross +3
Current large-scale auto-regressive language models display impressive fluency and can generate convincing text. In this work we start by asking the question: Can the generations o…
Facebook AI's WAT19 Myanmar-English Translation Task Submission
Peng-Jen Chen, Jiajun Shen, Matt Le +5
This paper describes Facebook AI's submission to the WAT 2019 Myanmar-English translation task. Our baseline systems are BPE-based transformer models. We explore methods to leverag…
The Source-Target Domain Mismatch Problem in Machine Translation
Jiajun Shen, Peng-Jen Chen, Matt Le +5
While we live in an increasingly interconnected world, different places still exhibit strikingly different cultures and many events we experience in our every day life pertain only…
On The Evaluation of Machine Translation Systems Trained With Back-Translation
Sergey Edunov, Myle Ott, Marc'Aurelio Ranzato +1
Back-translation is a widely used data augmentation technique which leverages target monolingual data. However, its effectiveness has been challenged since automatic metrics such a…
Large Memory Layers with Product Keys
Guillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato +2
This paper introduces a structured memory which can be easily integrated into a neural network. The memory is very large by design and significantly increases the capacity of the a…
Mixture Models for Diverse Machine Translation: Tricks of the Trade
Tianxiao Shen, Myle Ott, Michael Auli +1
Mixture models trained via EM are among the simplest, most widely used and well understood latent variable models in the machine learning literature. Surprisingly, these models hav…
The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English
Francisco Guzmán, Peng-Jen Chen, Myle Ott +5
For machine translation, a vast majority of language pairs in the world are considered low-resource because they have little parallel data available. Besides the technical challeng…
Multiple-Attribute Text Style Transfer
Sandeep Subramanian, Guillaume Lample, Eric Michael Smith +3
The dominant approach to unsupervised "style transfer" in text is based on the idea of learning a latent representation, which is independent of the attributes specifying its "styl…
Phrase-Based & Neural Unsupervised Machine Translation
Guillaume Lample, Myle Ott, Alexis Conneau +2
Machine translation systems achieve near human-level performance on some languages, yet their effectiveness strongly relies on the availability of large amounts of parallel sentenc…
Lightweight Adaptive Mixture of Neural and N-gram Language Models
Anton Bakhtin, Arthur Szlam, Marc'Aurelio Ranzato +1
It is often the case that the best performing language model is an ensemble of a neural language model with n-grams. In this work, we propose a method to improve how these two mode…
Analyzing Uncertainty in Neural Machine Translation
Myle Ott, Michael Auli, David Grangier +1
Machine translation is a popular test bed for research in neural sequence-to-sequence models but despite much recent research, there is still a lack of understanding of these model…
Classical Structured Prediction Losses for Sequence to Sequence Learning
Sergey Edunov, Myle Ott, Michael Auli +2
There has been much recent work on training neural attention models at the sequence-level using either reinforcement learning-style methods or by optimizing the beam. In this paper…
Unsupervised Machine Translation Using Monolingual Corpora Only
Guillaume Lample, Alexis Conneau, Ludovic Denoyer +1
Machine translation has recently achieved impressive performance thanks to recent advances in deep learning and the availability of large-scale parallel corpora. There have been nu…
Word Translation Without Parallel Data
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato +2
State-of-the-art methods for learning cross-lingual word embeddings have relied on bilingual dictionaries or parallel corpora. Recent studies showed that the need for parallel data…
Training Language Models Using Target-Propagation
Sam Wiseman, Sumit Chopra, Marc'Aurelio Ranzato +4
While Truncated Back-Propagation through Time (BPTT) is the most popular approach to training Recurrent Neural Networks (RNNs), it suffers from being inherently sequential (making…
Learning through Dialogue Interactions by Asking Questions
Jiwei Li, Alexander H. Miller, Sumit Chopra +2
A good dialogue agent should have the ability to interact with users by both responding to questions and by asking questions, and importantly to learn from both types of interactio…
Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews
Grégoire Mesnil, Tomas Mikolov, Marc'Aurelio Ranzato +1
Sentiment analysis is a common task in natural language processing that aims to detect polarity of a text document (typically a consumer review). In the simplest settings, we discr…