2.7k citations · 6.8k across the 64 of their papers we have counts for
74 papers
Structural Transfer Learning in NL-to-Bash Semantic Parsers
Kyle Duffy, Satwik Bhattamishra, Phil Blunsom
Large-scale pre-training has made progress in many fields of natural language processing, though little is understood about the design of pre-training datasets. We propose a method…
On "Scientific Debt" in NLP: A Case for More Rigour in Language Model Pre-Training Research
Made Nindyatama Nityasya, Haryo Akbarianto Wibowo, Alham Fikri Aji +4
This evidence-based position paper critiques current research practices within the language model pre-training literature. Despite rapid recent progress afforded by increasingly be…
Intriguing Properties of Quantization at Scale
Arash Ahmadian, Saurabh Dash, Hongyu Chen +5
Emergent properties have been widely adopted as a term to describe behavior not present in smaller models but observed in larger models. Recent work suggests that the trade-off inc…
Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions
Satwik Bhattamishra, Arkil Patel, Varun Kanade +1
Despite the widespread success of Transformers on NLP tasks, recent works have found that they struggle to model several formal languages when compared to recurrent models. This ra…
Augmenting Multi-Turn Text-to-SQL Datasets with Self-Play
Qi Liu, Zihuiwen Ye, Tao Yu +2
The task of context-dependent text-to-SQL aims to convert multi-turn user utterances to formal SQL queries. This is a challenging task due to both the scarcity of training data fro…
StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models
Adam Liška, Tomáš Kočiský, Elena Gribovskaya +11
Knowledge and language understanding of models evaluated through question answering (QA) has been usually studied on static snapshots of knowledge, like Wikipedia. However, our wor…
Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization
Aishwarya Agrawal, Ivana Kajić, Emanuele Bugliarello +4
Vision-and-language (V&L) models pretrained on large-scale multimodal data have demonstrated strong performance on various tasks such as image captioning and visual question answer…
Revisiting the Compositional Generalization Abilities of Neural Sequence Models
Arkil Patel, Satwik Bhattamishra, Phil Blunsom +1
Compositional generalization is a fundamental trait in humans, allowing us to effortlessly combine known phrases to form novel sentences. Recent works have claimed that standard se…
Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale
Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro +3
We introduce Transformer Grammars (TGs), a novel class of Transformer language models that combine (i) the expressive power, scalability, and strong performance of Transformers and…
Relational Memory Augmented Language Models
Qi Liu, Dani Yogatama, Phil Blunsom
We present a memory-augmented approach to condition an autoregressive language model on a knowledge graph. We represent the graph as a collection of relation triples and retrieve r…
A Systematic Investigation of Commonsense Knowledge in Large Language Models
Xiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoffmann +3
Language models (LMs) trained on large amounts of data have shown impressive performance on many NLP tasks under the zero-shot and few-shot setup. Here we aim to better understand…
Pretraining the Noisy Channel Model for Task-Oriented Dialogue
Qi Liu, Lei Yu, Laura Rimell +1
Direct decoding for task-oriented dialogue is known to suffer from the explaining-away effect, manifested in models that prefer short and generic responses. Here we argue for the u…
Mind the Gap: Assessing Temporal Generalization in Neural Language Models
Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya +11
Our world is open-ended, non-stationary, and constantly evolving; thus what we talk about and how we talk about it change over time. This inherent dynamic nature of language contra…
Mutual Information Constraints for Monte-Carlo Objectives
Gábor Melis, András György, Phil Blunsom
A common failure mode of density models trained as variational autoencoders is to model the data without relying on their latent variables, rendering these variables useless. Two c…
The Struggles of Feature-Based Explanations: Shapley Values vs. Minimal Sufficient Subsets
Oana-Maria Camburu, Eleonora Giunchiglia, Jakob Foerster +2
For neural models to garner widespread public trust and ensure fairness, we must have human-intelligible explanations for their predictions. Recently, an increasing number of works…
Syntactic Structure Distillation Pretraining For Bidirectional Encoders
Adhiguna Kuncoro, Lingpeng Kong, Daniel Fried +4
Textual representation learners trained on large amounts of data have achieved notable success on downstream tasks; intriguingly, they have also performed well on challenging tests…
Learning to Segment Actions from Observation and Narration
Daniel Fried, Jean-Baptiste Alayrac, Phil Blunsom +3
We apply a generative segmental model of task structure, guided by narration, to action segmentation in video. We focus on unsupervised and weakly-supervised settings where no acti…
A Survey on Contextual Embeddings
Qi Liu, Matt J. Kusner, Phil Blunsom
Contextual embeddings, such as ELMo and BERT, move beyond global word representations like Word2Vec and achieve ground-breaking performance on a wide range of natural language proc…
Visual Grounding in Video for Unsupervised Word Translation
Gunnar A. Sigurdsson, Jean-Baptiste Alayrac, Aida Nematzadeh +5
There are thousands of actively spoken languages on Earth, but a single visual world. Grounding in this visual world has the potential to bridge the gap between all these languages…
Learning Robust and Multilingual Speech Representations
Kazuya Kawakami, Luyu Wang, Chris Dyer +2
Unsupervised speech representation learning has shown remarkable success at finding representations that correlate with phonetic structures and improve downstream speech recognitio…
Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations
Oana-Maria Camburu, Brendan Shillingford, Pasquale Minervini +2
To increase trust in artificial intelligence systems, a promising research direction consists of designing neural models capable of generating natural language explanations for the…
Can I Trust the Explainer? Verifying Post-hoc Explanatory Methods
Oana-Maria Camburu, Eleonora Giunchiglia, Jakob Foerster +2
For AI systems to garner widespread public acceptance, we must develop methods capable of explaining the decisions of black-box models such as neural networks. In this work, we ide…
Better Document-Level Machine Translation with Bayes' Rule
Lei Yu, Laurent Sartran, Wojciech Stokowiec +4
We show that Bayes' rule provides an effective mechanism for creating document translation models that can be learned from only parallel sentences and monolingual documents---a com…
A Critical Analysis of Biased Parsers in Unsupervised Parsing
Chris Dyer, Gábor Melis, Phil Blunsom
A series of recent papers has used a parsing algorithm due to Shen et al. (2018) to recover phrase-structure trees based on proxies for "syntactic depth." These proxy depths are ob…