3.1k citations · 5k across the 42 of their papers we have counts for
57 papers
Human-Adversarial Visual Question Answering
Sasha Sheng, Amanpreet Singh, Vedanuj Goswami +4
Performance on the most commonly used Visual Question Answering dataset (VQA v2) is starting to approach human accuracy. However, in interacting with state-of-the-art VQA models, i…
On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study
Divyansh Kaushik, Douwe Kiela, Zachary C. Lipton +1
In adversarial data collection (ADC), a human workforce interacts with a model in real time, attempting to produce examples that elicit incorrect predictions. Researchers hope that…
True Few-Shot Learning with Language Models
Ethan Perez, Douwe Kiela, Kyunghyun Cho
Pretrained language models (LMs) perform well on many tasks even when learning from a few examples, but prior work uses many held-out examples to tune various aspects of learning,…
Cross-Modal Retrieval Augmentation for Multi-Modal Classification
Shir Gur, Natalia Neverova, Chris Stauffer +3
Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing. Here,…
Gradient-based Adversarial Attacks against Text Transformers
Chuan Guo, Alexandre Sablayrolles, Hervé Jégou +1
We propose the first general-purpose gradient-based attack against transformer models. Instead of searching for a single adversarial example, we search for a distribution of advers…
Retrieval Augmentation Reduces Hallucination in Conversation
Kurt Shuster, Spencer Poff, Moya Chen +2
Despite showing increasingly human-like conversational abilities, state-of-the-art dialogue models often suffer from factual incorrectness and hallucination of knowledge (Roller et…
Dynabench: Rethinking Benchmarking in NLP
Douwe Kiela, Max Bartolo, Yixin Nie +16
We introduce Dynabench, an open-source platform for dynamic dataset creation and model benchmarking. Dynabench runs in a web browser and supports human-and-model-in-the-loop datase…
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation
Max Bartolo, Tristan Thrush, Robin Jia +3
Despite recent progress, state-of-the-art question answering models remain vulnerable to a variety of adversarial attacks. While dynamic adversarial data collection, in which a hum…
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little
Koustuv Sinha, Robin Jia, Dieuwke Hupkes +3
A possible explanation for the impressive performance of masked language model (MLM) pre-training is that such models have learned to represent the syntactic structures prevalent i…
Rissanen Data Analysis: Examining Dataset Characteristics via Description Length
Ethan Perez, Douwe Kiela, Kyunghyun Cho
We introduce a method to determine if a certain capability helps to achieve an accurate model of given data. We view labels as being generated from the inputs by a program composed…
Quasi-Equivalence Discovery for Zero-Shot Emergent Communication
Kalesha Bullard, Douwe Kiela, Franziska Meier +2
Effective communication is an important skill for enabling information exchange in multi-agent settings and emergent communication is now a vibrant field of research, with common s…
DynaSent: A Dynamic Benchmark for Sentiment Analysis
Christopher Potts, Zhengxuan Wu, Atticus Geiger +1
We introduce DynaSent ('Dynamic Sentiment'), a new English-language benchmark task for ternary (positive/negative/neutral) sentiment analysis. DynaSent combines naturally occurring…
I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling
Yixin Nie, Mary Williamson, Mohit Bansal +2
To quantify how well natural language understanding models can capture consistency in a general conversation, we introduce the DialoguE COntradiction DEtection task (DECODE) and a…
Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection
Bertie Vidgen, Tristan Thrush, Zeerak Waseem +1
We present a human-and-model-in-the-loop process for dynamically generating datasets and training better performing and more robust hate detection models. We provide a new dataset…
Reservoir Transformers
Sheng Shen, Alexei Baevski, Ari S. Morcos +3
We demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. Inspired by old and well-established ideas i…
To what extent do human explanations of model behavior align with actual model behavior?
Grusha Prasad, Yixin Nie, Mohit Bansal +3
Given the increasingly prominent role NLP models (will) play in our lives, it is important for human expectations of model behavior to align with actual model behavior. Using Natur…
ANLIzing the Adversarial Natural Language Inference Dataset
Adina Williams, Tristan Thrush, Douwe Kiela
We perform an in-depth error analysis of Adversarial NLI (ANLI), a recently introduced large-scale human-and-model-in-the-loop natural language inference dataset collected over mul…
Exploring Zero-Shot Emergent Communication in Embodied Multi-Agent Populations
Kalesha Bullard, Franziska Meier, Douwe Kiela +2
Effective communication is an important skill for enabling information exchange and cooperation in multi-agent settings. Indeed, emergent communication is now a vibrant field of re…
Learning Optimal Representations with the Decodable Information Bottleneck
Yann Dubois, Douwe Kiela, David J. Schwab +1
We address the question of characterizing and finding optimal representations for supervised learning. Traditionally, this question has been tackled using the Information Bottlenec…
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
Wenhan Xiong, Xiang Lorraine Li, Srini Iyer +8
We propose a simple and efficient multi-hop dense retrieval approach for answering complex open-domain questions, which achieves state-of-the-art performance on two multi-hop datas…
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Patrick Lewis, Ethan Perez, Aleksandra Piktus +9
Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. Howe…
The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
Douwe Kiela, Hamed Firooz, Aravind Mohan +4
This work proposes a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes. It is constructed such that unimodal models struggle an…
Multi-Dimensional Gender Bias Classification
Emily Dinan, Angela Fan, Ledell Wu +3
Machine learning models are trained to find patterns in data. NLP models can inadvertently learn socially undesirable patterns when training on gender biased text. In this work, we…
I love your chain mail! Making knights smile in a fantasy game world: Open-domain goal-oriented dialogue agents
Shrimai Prabhumoye, Margaret Li, Jack Urbanek +4
Dialogue research tends to distinguish between chit-chat and goal-oriented tasks. While the former is arguably more naturalistic and has a wider use of language, the latter has cle…