Publications (50)
Non-Autoregressive Semantic Parsing for Compositional Task-Oriented Dialog
Arun Babu, Akshat Shrivastava, Armen Aghajanyan +3
Semantic parsing using sequence-to-sequence models allows parsing of deeper representations compared to traditional word tagging based models. In spite of these advantages, widespr…
Prior matters: simple and general methods for evaluating and improving topic quality in topic modeling
Angela Fan, Finale Doshi-Velez, Luke Miratrix
Latent Dirichlet Allocation (LDA) models trained without stopword removal often produce topics with high posterior probabilities on uninformative words, obscuring the underlying co…
Augmenting Transformers with KNN-Based Composite Memory for Dialogue
Angela Fan, Claire Gardent, Chloe Braud +1
Various machine learning tasks can benefit from access to external information of different modalities, such as text and images. Recent work has focused on learning architectures w…
Revisiting Machine Translation for Cross-lingual Classification
Mikel Artetxe, Vedanuj Goswami, Shruti Bhosale +2
Machine Translation (MT) has been widely used for cross-lingual classification, either by translating the test set into English and running inference with a monolingual model (tran…
Effective Long-Context Scaling of Foundation Models
Wenhan Xiong, Jingyu Liu, Igor Molybog +18
We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our model series are built through continual pretraining from Llama 2 with l…
Large Language Models for Software Engineering: Survey and Open Problems
Angela Fan, Beliz Gokkaya, Mark Harman +4
This paper provides a survey of the emerging area of Large Language Models (LLMs) for Software Engineering (SE). It also sets out open research challenges for the application of LL…
The Llama 3 Herd of Models
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri +556
Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models th…
Training with Quantization Noise for Extreme Model Compression
Angela Fan, Pierre Stock, Benjamin Graham +4
We tackle the problem of producing compact models, maximizing their accuracy for a given model size. A standard solution is to train networks with Quantization Aware Training, wher…
No Language Left Behind: Scaling Human-Centered Machine Translation
NLLB Team, Marta R. Costa-jussÃ, James Cross +36
Driven by the goal of eradicating language barriers on a global scale, machine translation has solidified itself as a key focus of artificial intelligence research today. However,…
Strategies for Structuring Story Generation
Angela Fan, Mike Lewis, Yann Dauphin
Writers generally rely on plans or sketches to write long stories, but most current language models generate word by word from left to right. We explore coarse-to-fine models for c…
Generating Full Length Wikipedia Biographies: The Impact of Gender Bias on the Retrieval-Based Generation of Women Biographies
Angela Fan, Claire Gardent
Generating factual, long-form text such as Wikipedia articles raises three key challenges: how to gather relevant evidence, how to structure information into well-formed text, and…
Facebook AI's WMT20 News Translation Task Submission
Peng-Jen Chen, Ann Lee, Changhan Wang +4
This paper describes Facebook AI's submission to WMT20 shared news translation task. We focus on the low resource setting and participate in two language pairs, Tamil <-> English a…
Controllable Abstractive Summarization
Angela Fan, David Grangier, Michael Auli
Current models for document summarization disregard user preferences such as the desired length, style, the entities that the user might be interested in, or how much of the docume…
AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages
Abteen Ebrahimi, Manuel Mager, Arturo Oncevay +14
Pretrained multilingual models are able to perform cross-lingual transfer in a zero-shot setting, even for languages unseen during pretraining. However, prior work evaluating perfo…
Alternative Input Signals Ease Transfer in Multilingual Machine Translation
Simeng Sun, Angela Fan, James Cross +4
Recent work in multilingual machine translation (MMT) has focused on the potential of positive transfer between languages, particularly cases where higher-resourced languages can b…
Not All Memories are Created Equal: Learning to Forget by Expiring
Sainbayar Sukhbaatar, Da Ju, Spencer Poff +4
Attention mechanisms have shown promising results in sequence modeling tasks that require long-term memory. Recent work investigated mechanisms to reduce the computational cost of…
MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases
Louis Martin, Angela Fan, Ãric de la Clergerie +2
Progress in sentence simplification has been hindered by a lack of labeled parallel simplification data, particularly in languages other than English. We introduce MUSS, a Multilin…
Ngambay-French Neural Machine Translation (sba-Fr)
Sakayo Toadoum Sari, Angela Fan, Lema Logamou Seknewna
In Africa, and the world at large, there is an increasing focus on developing Neural Machine Translation (NMT) systems to overcome language barriers. NMT for Low-resource language…
Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QA
Ana Valeria Gonzalez, Gagan Bansal, Angela Fan +3
While research on explaining predictions of open-domain QA systems (ODQA) to users is gaining momentum, most works have failed to evaluate the extent to which explanations improve…
Facebook AI WMT21 News Translation Task Submission
Chau Tran, Shruti Bhosale, James Cross +3
We describe Facebook's multilingual model submission to the WMT2021 shared task on news translation. We participate in 14 language directions: English to and from Czech, German, Ha…
Reducing Transformer Depth on Demand with Structured Dropout
Angela Fan, Edouard Grave, Armand Joulin
Overparameterized transformer networks have obtained state of the art results in various natural language processing tasks, such as machine translation, language modeling, and ques…
ELI5: Long Form Question Answering
Angela Fan, Yacine Jernite, Ethan Perez +3
We introduce the first large-scale corpus for long-form question answering, a task requiring elaborate and in-depth answers to open-ended questions. The dataset comprises 270K thre…
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
BigScience Workshop, :, Teven Le Scao +391
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to wi…
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Myle Ott, Sergey Edunov, Alexei Baevski +5
fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text…
Generating Interactive Worlds with Text
Angela Fan, Jack Urbanek, Pratik Ringshia +8
Procedurally generating cohesive and interesting game environments is challenging and time-consuming. In order for the relationships between the game elements to be natural, common…
Multilingual AMR-to-Text Generation
Angela Fan, Claire Gardent
Generating text from structured data is challenging because it requires bridging the gap between (i) structure and natural language (NL) and (ii) semantically underspecified input…
Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation
Emily Dinan, Angela Fan, Adina Williams +3
Models often easily learn biases present in the training data, and their predictions directly reflect this bias. We analyze gender bias in dialogue data, and examine how this bias…
Learning to Speak and Act in a Fantasy Text Adventure Game
Jack Urbanek, Angela Fan, Siddharth Karamcheti +7
We introduce a large scale crowdsourced text adventure game as a research platform for studying grounded dialogue. In it, agents can perceive, emote, and act whilst conducting dial…
Nearest Neighbor Machine Translation
Urvashi Khandelwal, Angela Fan, Dan Jurafsky +2
We introduce -nearest-neighbor machine translation (NN-MT), which predicts tokens with a nearest neighbor classifier over a large datastore of cached examples, using represen…
Addressing Some Limitations of Transformers with Feedback Memory
Angela Fan, Thibaut Lavril, Edouard Grave +2
Transformers have been successfully applied to sequential, auto-regressive tasks despite being feedforward networks. Unlike recurrent neural networks, Transformers use attention to…
Hierarchical Neural Story Generation
Angela Fan, Mike Lewis, Yann Dauphin
We explore story generation: creative systems that can build coherent and fluent passages of text about a topic. We collect a large dataset of 300K human-written stories paired wit…
Reasoning over Public and Private Data in Retrieval-Based Systems
Simran Arora, Patrick Lewis, Angela Fan +2
Users and organizations are generating ever-increasing amounts of private data from a wide range of sources. Incorporating private data is important to personalize open-domain appl…
A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation
David Ifeoluwa Adelani, Jesujoba Oluwadara Alabi, Angela Fan +42
Recent advances in the pre-training of language models leverage large-scale datasets to create multilingual models. However, low-resource languages are mostly left out in these dat…
Using Local Knowledge Graph Construction to Scale Seq2Seq Models to Multi-Document Inputs
Angela Fan, Claire Gardent, Chloe Braud +1
Query-based open-domain NLP tasks require information synthesis from long and diverse web results. Current approaches extractively select portions of web text as input to Sequence-…
KILT: a Benchmark for Knowledge Intensive Language Tasks
Fabio Petroni, Aleksandra Piktus, Angela Fan +10
Challenging problems such as open-domain question answering, fact checking, slot filling and entity linking require access to large, external knowledge sources. While some models d…
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions
Stephen Roller, Y-Lan Boureau, Jason Weston +13
We present our view of what is necessary to build an engaging open-domain conversational agent: covering the qualities of such an agent, the pieces of the puzzle that have been bui…
Beyond English-Centric Multilingual Machine Translation
Angela Fan, Shruti Bhosale, Holger Schwenk +14
Existing work in translation demonstrated the potential of massively multilingual machine translation by training a single model able to translate between any pair of languages. Ho…
Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
Yuqing Tang, Chau Tran, Xian Li +5
Recent work demonstrates the potential of multilingual pretraining of creating one model that can be used for various tasks in different languages. Previous work in multilingual pr…
Tricks for Training Sparse Translation Models
Dheeru Dua, Shruti Bhosale, Vedanuj Goswami +3
Multi-task learning with an unbalanced data distribution skews model learning towards high resource tasks, especially when model capacity is fixed and fully shared across all tasks…
GLOSS: Generative Latent Optimization of Sentence Representations
Sidak Pal Singh, Angela Fan, Michael Auli
We propose a method to learn unsupervised sentence representations in a non-compositional manner based on Generative Latent Optimization. Our approach does not impose any assumptio…
Generating Fact Checking Briefs
Angela Fan, Aleksandra Piktus, Fabio Petroni +5
Fact checking at scale is difficult -- while the number of active fact checking websites is growing, it remains too small for the needs of the contemporary media ecosystem. However…
CycleDRUMS: Automatic Drum Arrangement For Bass Lines Using CycleGAN
Giorgio Barnabò, Giovanni Trappolini, Lorenzo Lastilla +4
The two main research threads in computer-based music generation are: the construction of autonomous music-making systems, and the design of computer-based environments to assist m…
Wizard of Wikipedia: Knowledge-Powered Conversational agents
Emily Dinan, Stephen Roller, Kurt Shuster +3
In open-domain dialogue intelligent agents should exhibit the use of knowledge, however there are few convincing demonstrations of this to date. The most popular sequence to sequen…
The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation
Naman Goyal, Cynthia Gao, Vishrav Chaudhary +7
One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks. Current evaluation benchmarks eithe…
Llama 2: Open Foundation and Fine-Tuned Chat Models
Hugo Touvron, Louis Martin, Kevin Stone +65
In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our f…
Pay Less Attention with Lightweight and Dynamic Convolutions
Felix Wu, Angela Fan, Alexei Baevski +2
Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current t…
RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question
Alireza Mohammadshahi, Thomas Scialom, Majid Yazdani +4
Existing metrics for evaluating the quality of automatically generated questions such as BLEU, ROUGE, BERTScore, and BLEURT compare the reference and predicted questions, providing…
Language Modeling with Gated Convolutional Networks
Yann N. Dauphin, Angela Fan, Michael Auli +1
The pre-dominant approach to language modeling to date is based on recurrent neural networks. Their success on this task is often linked to their ability to capture unbounded conte…
Multi-Dimensional Gender Bias Classification
Emily Dinan, Angela Fan, Ledell Wu +3
Machine learning models are trained to find patterns in data. NLP models can inadvertently learn socially undesirable patterns when training on gender biased text. In this work, we…
Stress Testing Deliberative Alignment for Anti-Scheming Training
Bronson Schoen, Evgenia Nitishinskaya, Mikita Balesni +16
Highly capable AI systems could secretly pursue misaligned goals -- what we call "scheming". Because a scheming AI would deliberately try to hide its misaligned goals and actions,…