70 citations · 239 across the 24 of their papers we have counts for
46 papers
Towards a Similarity-adjusted Surprisal Theory
Clara Meister, Mario Giulianelli, Tiago Pimentel
Surprisal theory posits that the cognitive effort required to comprehend a word is determined by its contextual predictability, quantified as surprisal. Traditionally, surprisal th…
Investigating Critical Period Effects in Language Acquisition through Neural Language Models
Ionut Constantinescu, Tiago Pimentel, Ryan Cotterell +1
Humans appear to have a critical period (CP) for language acquisition: Second language (L2) acquisition becomes harder after early childhood, and ceasing exposure to a first langua…
How to Compute the Probability of a Word
Tiago Pimentel, Clara Meister
Language models (LMs) estimate a probability distribution over strings in a natural language; these distributions are crucial for computing perplexity and surprisal in linguistics…
Revisiting the Optimality of Word Lengths
Tiago Pimentel, Clara Meister, Ethan Gotlieb Wilcox +2
Zipf (1935) posited that wordforms are optimized to minimize utterances' communicative costs. Under the assumption that cost is given by an utterance's length, he supported this cl…
Quantifying the redundancy between prosody and text
Lukas Wolf, Tiago Pimentel, Evelina Fedorenko +4
Prosody -- the suprasegmental component of speech, including pitch, loudness, and tempo -- carries critical aspects of meaning. However, the relationship between the information co…
An Exploration of Left-Corner Transformations
Andreas Opedal, Eleftheria Tsipidi, Tiago Pimentel +2
The left-corner transformation (Rosenkrantz and Lewis, 1970) is used to remove left recursion from context-free grammars, which is an important step towards making the grammar pars…
A Cross-Linguistic Pressure for Uniform Information Density in Word Order
Thomas Hikaru Clark, Clara Meister, Tiago Pimentel +4
While natural languages differ widely in both canonical word order and word order flexibility, their word orders still follow shared cross-linguistic statistical patterns, often at…
On the Efficacy of Sampling Adapters
Clara Meister, Tiago Pimentel, Luca Malagutti +2
Sampling is a common strategy for generating text from probabilistic models, yet standard ancestral sampling often results in text that is incoherent or ungrammatical. To alleviate…
Testing the Predictions of Surprisal Theory in 11 Languages
Ethan Gotlieb Wilcox, Tiago Pimentel, Clara Meister +2
A fundamental result in psycholinguistics is that less predictable words take a longer time to process. One theoretical explanation for this finding is Surprisal Theory (Hale, 2001…
Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation
Marius Mosbach, Tiago Pimentel, Shauli Ravfogel +2
Few-shot fine-tuning and in-context learning are two alternative strategies for task adaptation of pre-trained language models. Recently, in-context learning has gained popularity…
A Measure-Theoretic Characterization of Tight Language Models
Li Du, Lucas Torroba Hennigen, Tiago Pimentel +3
Language modeling, a central task in natural language processing, involves estimating a probability distribution over strings. In most cases, the estimated distribution sums to 1 o…
A Natural Bias for Language Generation Models
Clara Meister, Wojciech Stokowiec, Tiago Pimentel +3
After just a few hundred training updates, a standard probabilistic model for language generation has likely not yet learnt many semantic or syntactic rules of natural language, ma…
The Architectural Bottleneck Principle
Tiago Pimentel, Josef Valvoda, Niklas Stoehr +1
In this paper, we seek to measure how much information a component in a neural network could extract from the representations fed into it. Our work stands in contrast to prior prob…
On the Effect of Anticipation on Reading Times
Tiago Pimentel, Clara Meister, Ethan G. Wilcox +2
Over the past two decades, numerous studies have demonstrated how less predictable (i.e., higher surprisal) words take more time to read. In general, these studies have implicitly…
State-of-the-art generalisation research in NLP: A taxonomy and review
Dieuwke Hupkes, Mario Giulianelli, Verna Dankers +17
The ability to generalise well is one of the primary desiderata of natural language processing (NLP). Yet, what 'good generalisation' entails and how it should be evaluated is not…
On the Intersection of Context-Free and Regular Languages
Clemente Pasti, Andreas Opedal, Tiago Pimentel +3
The Bar-Hillel construction is a classic result in formal language theory. It shows, by a simple construction, that the intersection of a context-free language and a regular langua…
Rethinking Reinforcement Learning for Recommendation: A Prompt Perspective
Xin Xin, Tiago Pimentel, Alexandros Karatzoglou +3
Modern recommender systems aim to improve user experience. As reinforcement learning (RL) naturally fits this objective -- maximizing an user's reward per session -- it has become…
On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluation
Tiago Pimentel, Clara Meister, Ryan Cotterell
A good automatic evaluation metric for language generation ideally correlates highly with human judgements of text quality. Yet, there is a dearth of such metrics, which inhibits t…
Naturalistic Causal Probing for Morpho-Syntax
Afra Amini, Tiago Pimentel, Clara Meister +1
Probing has become a go-to methodology for interpreting and analyzing deep neural models in natural language processing. However, there is still a lack of understanding of the limi…
Probing for the Usage of Grammatical Number
Karim Lasri, Tiago Pimentel, Alessandro Lenci +2
A central quest of probing is to uncover how pre-trained models encode a linguistic property within their representations. An encoding, however, might be spurious-i.e., the model m…
On the probability-quality paradox in language generation
Clara Meister, Gian Wiher, Tiago Pimentel +1
When generating natural language from neural probabilistic models, high probability does not always coincide with high quality: It has often been observed that mode-seeking decodin…
Analyzing Wrap-Up Effects through an Information-Theoretic Lens
Clara Meister, Tiago Pimentel, Thomas Hikaru Clark +2
Numerous analyses of reading time (RT) data have been implemented -- all in an effort to better understand the cognitive processes driving reading comprehension. However, data meas…
Locally Typical Sampling
Clara Meister, Tiago Pimentel, Gian Wiher +1
Today's probabilistic language generators fall short when it comes to producing coherent and fluent text despite the fact that the underlying models perform well under standard met…
A surprisal--duration trade-off across and within the world's languages
Tiago Pimentel, Clara Meister, Elizabeth Salesky +3
While there exist scores of natural languages, each with its unique features and idiosyncrasies, they all share a unifying theme: enabling human communication. We may thus reasonab…