72 citations · 243 across the 30 of their papers we have counts for
15 papers · 1 filter
A Measure-Theoretic Characterization of Tight Language Models
Li Du, Lucas Torroba Hennigen, Tiago Pimentel +3
Language modeling, a central task in natural language processing, involves estimating a probability distribution over strings. In most cases, the estimated distribution sums to 1 o…
Schrödinger's Bat: Diffusion Models Sometimes Generate Polysemous Words in Superposition
Jennifer C. White, Ryan Cotterell
Recent work has shown that despite their impressive capabilities, text-to-image diffusion models such as DALL-E 2 (Ramesh et al., 2022) can display strange behaviours when a prompt…
The Architectural Bottleneck Principle
Tiago Pimentel, Josef Valvoda, Niklas Stoehr +1
In this paper, we seek to measure how much information a component in a neural network could extract from the representations fed into it. Our work stands in contrast to prior prob…
On the Effect of Anticipation on Reading Times
Tiago Pimentel, Clara Meister, Ethan G. Wilcox +2
Over the past two decades, numerous studies have demonstrated how less predictable (i.e., higher surprisal) words take more time to read. In general, these studies have implicitly…
Mutual Information Alleviates Hallucinations in Abstractive Summarization
Liam van der Poel, Ryan Cotterell, Clara Meister
Despite significant progress in the quality of language generated from abstractive summarization models, these models still exhibit the tendency to hallucinate, i.e., output conten…
State-of-the-art generalisation research in NLP: A taxonomy and review
Dieuwke Hupkes, Mario Giulianelli, Verna Dankers +17
The ability to generalise well is one of the primary desiderata of natural language processing (NLP). Yet, what 'good generalisation' entails and how it should be evaluated is not…
On the Intersection of Context-Free and Regular Languages
Clemente Pasti, Andreas Opedal, Tiago Pimentel +3
The Bar-Hillel construction is a classic result in formal language theory. It shows, by a simple construction, that the intersection of a context-free language and a regular langua…
On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluation
Tiago Pimentel, Clara Meister, Ryan Cotterell
A good automatic evaluation metric for language generation ideally correlates highly with human judgements of text quality. Yet, there is a dearth of such metrics, which inhibits t…
Naturalistic Causal Probing for Morpho-Syntax
Afra Amini, Tiago Pimentel, Clara Meister +1
Probing has become a go-to methodology for interpreting and analyzing deep neural models in natural language processing. However, there is still a lack of understanding of the limi…
Estimating the Entropy of Linguistic Distributions
Aryaman Arora, Clara Meister, Ryan Cotterell
Shannon entropy is often a quantity of interest to linguists studying the communicative capacity of human language. However, entropy must typically be estimated from observed data…
Probing for the Usage of Grammatical Number
Karim Lasri, Tiago Pimentel, Alessandro Lenci +2
A central quest of probing is to uncover how pre-trained models encode a linguistic property within their representations. An encoding, however, might be spurious-i.e., the model m…
On the probability-quality paradox in language generation
Clara Meister, Gian Wiher, Tiago Pimentel +1
When generating natural language from neural probabilistic models, high probability does not always coincide with high quality: It has often been observed that mode-seeking decodin…
On Decoding Strategies for Neural Text Generators
Gian Wiher, Clara Meister, Ryan Cotterell
When generating text from probabilistic models, the chosen decoding strategy has a profound effect on the resulting text. Yet the properties elicited by various decoding strategies…
Analyzing Wrap-Up Effects through an Information-Theoretic Lens
Clara Meister, Tiago Pimentel, Thomas Hikaru Clark +2
Numerous analyses of reading time (RT) data have been implemented -- all in an effort to better understand the cognitive processes driving reading comprehension. However, data meas…
Locally Typical Sampling
Clara Meister, Tiago Pimentel, Gian Wiher +1
Today's probabilistic language generators fall short when it comes to producing coherent and fluent text despite the fact that the underlying models perform well under standard met…