343 citations · 489 across the 23 of their papers we have counts for
17 papers · 1 filter
Shared Circuits for Shared Grammar: Tracing Subject-Verb Agreement Across Languages
Isabella Gidi, Antonio Almudévar, Core Francisco Park +2
Multilingual large language models often generalize across languages, and prior work suggests that their internal mechanisms can overlap cross-lingually. It remains unclear, howeve…
Do Activation Verbalization Methods Convey Privileged Information?
Millicent Li, Alberto Mario Ceballos Arroyo, Giordano Rogers +2
Recent interpretability methods have proposed to translate LLM internal representations into natural language descriptions using a second verbalizer LLM. This is intended to illumi…
PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs
Oskar van der Wal, Pietro Lesci, Max Muller-Eberstein +4
The stability of language model pre-training and its effects on downstream performance are still understudied. Prior work shows that the training process can yield significantly di…
Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon
USVSN Sai Prashanth, Alvin Deng, Kyle O'Brien +9
Memorization in language models is typically treated as a homogenous phenomenon, neglecting the specifics of the memorized data. We instead model memorization as the effect of a se…
First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models
Naomi Saphra, Eve Fleisig, Kyunghyun Cho +1
Many NLP researchers are experiencing an existential crisis triggered by the astonishing success of ChatGPT and other systems based on large language models (LLMs). After such a di…
Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs
Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho +2
Most interpretability research in NLP focuses on understanding the behavior and features of a fully trained model. However, certain insights into model behavior may only be accessi…
Dynamic Masking Rate Schedules for MLM Pretraining
Zachary Ankner, Naomi Saphra, Davis Blalock +2
Most works on transformers trained with the Masked Language Modeling (MLM) objective use the original BERT model's fixed masking rate of 15%. We propose to instead dynamically sche…
State-of-the-art generalisation research in NLP: A taxonomy and review
Dieuwke Hupkes, Mario Giulianelli, Verna Dankers +17
The ability to generalise well is one of the primary desiderata of natural language processing (NLP). Yet, what 'good generalisation' entails and how it should be evaluated is not…
Benchmarking Compositionality with Formal Languages
Josef Valvoda, Naomi Saphra, Jonathan Rawski +2
Recombining known primitive concepts into larger novel combinations is a quintessentially human cognitive capability. Whether large neural models in NLP can acquire this ability wh…
The MultiBERTs: BERT Reproductions for Robustness Analysis
Thibault Sellam, Steve Yadlowsky, Jason Wei +9
Experiments with pre-trained models such as BERT are often based on a single checkpoint. While the conclusions drawn apply to the artifact tested in the experiment (i.e., the parti…
A Non-Linear Structural Probe
Jennifer C. White, Tiago Pimentel, Naomi Saphra +1
Probes are models devised to investigate the encoding of knowledge -- e.g. syntactic structure -- in contextual representations. Probes are often designed for simplicity, which has…
LSTMs Compose (and Learn) Bottom-Up
Naomi Saphra, Adam Lopez
Recent work in NLP shows that LSTM language models capture hierarchical structure in language data. In contrast to existing work, we consider the \textit{learning} process that lea…
Pareto Probing: Trading Off Accuracy for Complexity
Tiago Pimentel, Naomi Saphra, Adina Williams +1
The question of how to probe contextual word representations for linguistic structure in a way that is both principled and useful has seen significant attention recently in the NLP…
Word Interdependence Exposes How LSTMs Compose Representations
Naomi Saphra, Adam Lopez
Recent work in NLP shows that LSTM language models capture compositional structure in language data. For a closer look at how these representations are composed hierarchically, we…
Sparsity Emerges Naturally in Neural Language Models
Naomi Saphra, Adam Lopez
Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks. If sparsity is impo…
Evaluating Informal-Domain Word Representations With UrbanDictionary
Naomi Saphra, Adam Lopez
Existing corpora for intrinsic evaluation are not targeted towards tasks in informal domains such as Twitter or news comment forums. We want to test whether a representation of inf…
A framework for (under)specifying dependency syntax without overloading annotators
Nathan Schneider, Brendan O'Connor, Naomi Saphra +5
We introduce a framework for lightweight dependency syntax annotation. Our formalism builds upon the typical representation for unlabeled dependencies, permitting a simple notation…