72 citations · 243 across the 30 of their papers we have counts for
52 papers
A Formal Limitation on Learning Human Language From Textual Corpora
Emily Cheng, Ryan Cotterell
Can a listener recover what a speaker means from the form of an utterance alone? We answer this question information-theoretically, and for a listener given by any featurizer of te…
Surprisal Theory is Tautological (without Rational Grounding)
Ryan Cotterell
Surprisal theory holds that the human processing difficulty of a linguistic unit in context is an affine function of its surprisal under some language model. I argue this claim is…
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Alex Warstadt, Aaron Mueller, Leshem Choshen +8
Children can acquire language from less than 100 million words of input. Large language models are far less data-efficient: they typically require 3 or 4 orders of magnitude more d…
Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Michael Y. Hu, Aaron Mueller, Candace Ross +7
The BabyLM Challenge is a community effort to close the data-efficiency gap between human and computational language learners. Participants compete to optimize language model train…
Surprise! Uniform Information Density Isn't the Whole Story: Predicting Surprisal Contours in Long-form Discourse
Eleftheria Tsipidi, Franz Nowak, Ryan Cotterell +3
The Uniform Information Density (UID) hypothesis posits that speakers tend to distribute information evenly across linguistic units to achieve efficient communication. Of course, i…
Reverse-Engineering the Reader
Samuel Kiegeland, Ethan Gotlieb Wilcox, Afra Amini +2
Numerous previous studies have sought to determine to what extent language models, pretrained on natural language text, can serve as useful models of human cognition. In this paper…
Generalized Measures of Anticipation and Responsivity in Online Language Processing
Mario Giulianelli, Andreas Opedal, Ryan Cotterell
We introduce a generalization of classic information-theoretic measures of predictive uncertainty in online language processing, based on the simulation of expected continuations o…
Thesis proposal: Are We Losing Textual Diversity to Natural Language Processing?
Josef Jon
This thesis argues that the currently widely used Natural Language Processing algorithms possibly have various limitations related to the properties of the texts they handle and pr…
On the Role of Context in Reading Time Prediction
Andreas Opedal, Eleanor Chodroff, Ryan Cotterell +1
We present a new perspective on how readers integrate context during real-time language comprehension. Our proposals build on surprisal theory, which posits that the processing eff…
Investigating Critical Period Effects in Language Acquisition through Neural Language Models
Ionut Constantinescu, Tiago Pimentel, Ryan Cotterell +1
Humans appear to have a critical period (CP) for language acquisition: Second language (L2) acquisition becomes harder after early childhood, and ceasing exposure to a first langua…
The Role of -gram Smoothing in the Age of Neural Networks
Luca Malagutti, Andrius Buinovskij, Anej Svete +3
For nearly three decades, language models derived from the -gram assumption held the state of the art on the task. The key to their success lay in the application of various smo…
[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
Leshem Choshen, Ryan Cotterell, Michael Y. Hu +7
After last year's successful BabyLM Challenge, the competition will be hosted again in 2024/2025. The overarching goals of the challenge remain the same; however, some of the compe…
Revisiting the Optimality of Word Lengths
Tiago Pimentel, Clara Meister, Ethan Gotlieb Wilcox +2
Zipf (1935) posited that wordforms are optimized to minimize utterances' communicative costs. Under the assumption that cost is given by an utterance's length, he supported this cl…
Quantifying the redundancy between prosody and text
Lukas Wolf, Tiago Pimentel, Evelina Fedorenko +4
Prosody -- the suprasegmental component of speech, including pitch, loudness, and tempo -- carries critical aspects of meaning. However, the relationship between the information co…
An Exploration of Left-Corner Transformations
Andreas Opedal, Eleftheria Tsipidi, Tiago Pimentel +2
The left-corner transformation (Rosenkrantz and Lewis, 1970) is used to remove left recursion from context-free grammars, which is an important step towards making the grammar pars…
The Causal Influence of Grammatical Gender on Distributional Semantics
Karolina Stańczak, Kevin Du, Adina Williams +2
How much meaning influences gender assignment across languages is an active area of research in linguistics and cognitive science. We can view current approaches as aiming to deter…
Formal Aspects of Language Modeling
Ryan Cotterell, Anej Svete, Clara Meister +2
Large language models have become one of the most commonly deployed NLP inventions. In the past half-decade, their integration into core natural language processing tools has drama…
A Cross-Linguistic Pressure for Uniform Information Density in Word Order
Thomas Hikaru Clark, Clara Meister, Tiago Pimentel +4
While natural languages differ widely in both canonical word order and word order flexibility, their word orders still follow shared cross-linguistic statistical patterns, often at…
On the Efficacy of Sampling Adapters
Clara Meister, Tiago Pimentel, Luca Malagutti +2
Sampling is a common strategy for generating text from probabilistic models, yet standard ancestral sampling often results in text that is incoherent or ungrammatical. To alleviate…
Testing the Predictions of Surprisal Theory in 11 Languages
Ethan Gotlieb Wilcox, Tiago Pimentel, Clara Meister +2
A fundamental result in psycholinguistics is that less predictable words take a longer time to process. One theoretical explanation for this finding is Surprisal Theory (Hale, 2001…
A Measure-Theoretic Characterization of Tight Language Models
Li Du, Lucas Torroba Hennigen, Tiago Pimentel +3
Language modeling, a central task in natural language processing, involves estimating a probability distribution over strings. In most cases, the estimated distribution sums to 1 o…
Schrödinger's Bat: Diffusion Models Sometimes Generate Polysemous Words in Superposition
Jennifer C. White, Ryan Cotterell
Recent work has shown that despite their impressive capabilities, text-to-image diffusion models such as DALL-E 2 (Ramesh et al., 2022) can display strange behaviours when a prompt…
The Architectural Bottleneck Principle
Tiago Pimentel, Josef Valvoda, Niklas Stoehr +1
In this paper, we seek to measure how much information a component in a neural network could extract from the representations fed into it. Our work stands in contrast to prior prob…
On the Effect of Anticipation on Reading Times
Tiago Pimentel, Clara Meister, Ethan G. Wilcox +2
Over the past two decades, numerous studies have demonstrated how less predictable (i.e., higher surprisal) words take more time to read. In general, these studies have implicitly…