242 citations · 1.5k across the 60 of their papers we have counts for
7 papers · 2 filters
Quantifying Adaptability in Pre-trained Language Models with 500 Tasks
Belinda Z. Li, Jane Yu, Madian Khabsa +3
When a neural language model (LM) is adapted to perform a new task, what aspects of the task predict the eventual performance of the model? In NLP, systematic features of LM genera…
FairyTailor: A Multimodal Generative Framework for Storytelling
Eden Bensaid, Mauro Martino, Benjamin Hoover +1
Storytelling is an open-ended task that entails creative thinking and requires a constant flow of ideas. Natural language generation (NLG) for storytelling is especially challengin…
What Context Features Can Transformer Language Models Use?
Joe O'Connor, Jacob Andreas
Transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. What aspects of these contexts contribute to accurate model pre…
Lexicon Learning for Few-Shot Neural Sequence Modeling
Ekin Akyürek, Jacob Andreas
Sequence-to-sequence transduction is the core problem in language processing applications as diverse as semantic parsing, machine translation, and instruction following. The neural…
Implicit Representations of Meaning in Neural Language Models
Belinda Z. Li, Maxwell Nye, Jacob Andreas
Does the effectiveness of neural language models derive entirely from accurate modeling of surface word co-occurrence statistics, or do these models represent and reason about the…
The Low-Dimensional Linear Geometry of Contextualized Word Representations
Evan Hernandez, Jacob Andreas
Black-box probing models can reliably extract linguistic features like tense, number, and syntactic role from pretrained word representations. However, the manner in which these fe…
Multitasking Inhibits Semantic Drift
Athul Paul Jacob, Mike Lewis, Jacob Andreas
When intelligent agents communicate to accomplish shared goals, how do these goals shape the agents' language? We study the dynamics of learning in latent language policies (LLPs),…