11 papers
Surprisal Theory is Tautological (without Rational Grounding)
Ryan Cotterell
Surprisal theory holds that the human processing difficulty of a linguistic unit in context is an affine function of its surprisal under some language model. I argue this claim is…
On the Role of Context in Reading Time Prediction
Andreas Opedal, Eleanor Chodroff, Ryan Cotterell +1
We present a new perspective on how readers integrate context during real-time language comprehension. Our proposals build on surprisal theory, which posits that the processing eff…
Locally Typical Sampling
Clara Meister, Tiago Pimentel, Gian Wiher +1
Today's probabilistic language generators fall short when it comes to producing coherent and fluent text despite the fact that the underlying models perform well under standard met…
Testing the Predictions of Surprisal Theory in 11 Languages
Ethan Gotlieb Wilcox, Tiago Pimentel, Clara Meister +2
A fundamental result in psycholinguistics is that less predictable words take a longer time to process. One theoretical explanation for this finding is Surprisal Theory (Hale, 2001…
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Alex Warstadt, Aaron Mueller, Leshem Choshen +8
Children can acquire language from less than 100 million words of input. Large language models are far less data-efficient: they typically require 3 or 4 orders of magnitude more d…
Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Michael Y. Hu, Aaron Mueller, Candace Ross +7
The BabyLM Challenge is a community effort to close the data-efficiency gap between human and computational language learners. Participants compete to optimize language model train…