72 citations · 243 across the 30 of their papers we have counts for
16 papers · 1 filter
A surprisal--duration trade-off across and within the world's languages
Tiago Pimentel, Clara Meister, Elizabeth Salesky +3
While there exist scores of natural languages, each with its unique features and idiosyncrasies, they all share a unifying theme: enabling human communication. We may thus reasonab…
On Homophony and Rényi Entropy
Tiago Pimentel, Clara Meister, Simone Teufel +1
Homophony's widespread presence in natural languages is a controversial topic. Recent theories of language optimality have tried to justify its prevalence, despite its negative eff…
Revisiting the Uniform Information Density Hypothesis
Clara Meister, Tiago Pimentel, Patrick Haller +3
The uniform information density (UID) hypothesis posits a preference among language users for utterances structured such that information is distributed uniformly across a signal.…
A Bayesian Framework for Information-Theoretic Probing
Tiago Pimentel, Ryan Cotterell
Pimentel et al. (2020) recently analysed probing from an information-theoretic perspective. They argue that probing should be seen as approximating a mutual information. This led t…
Conditional Poisson Stochastic Beam Search
Clara Meister, Afra Amini, Tim Vieira +1
Beam search is the default decoding strategy for many sequence generation tasks in NLP. The set of approximate K-best items returned by the algorithm is a useful summary of the dis…
A Cognitive Regularizer for Language Modeling
Jason Wei, Clara Meister, Ryan Cotterell
The uniform information density (UID) hypothesis, which posits that speakers behaving optimally tend to distribute information uniformly across a linguistic signal, has gained trac…
Is Sparse Attention more Interpretable?
Clara Meister, Stefan Lazov, Isabelle Augenstein +1
Sparse attention has been claimed to increase model interpretability under the assumption that it highlights influential inputs. Yet the attention distribution is typically over re…
Modeling the Unigram Distribution
Irene Nikkarinen, Tiago Pimentel, Damián E. Blasi +1
The unigram distribution is the non-contextual probability of finding a specific word form in a corpus. While of central importance to the study of language, it is commonly approxi…
Examining the Inductive Bias of Neural Language Models with Artificial Languages
Jennifer C. White, Ryan Cotterell
Since language models are used to model a wide variety of languages, it is natural to ask whether the neural architectures used for the task have inductive biases towards modeling…
Determinantal Beam Search
Clara Meister, Martina Forster, Ryan Cotterell
Beam search is a go-to strategy for decoding neural sequence models. The algorithm can naturally be viewed as a subset optimization problem, albeit one where the corresponding set…
Language Model Evaluation Beyond Perplexity
Clara Meister, Ryan Cotterell
We propose an alternate approach to quantifying how well language models learn natural language: we ask how well they match the statistical tendencies of natural language. To answe…
A Non-Linear Structural Probe
Jennifer C. White, Tiago Pimentel, Naomi Saphra +1
Probes are models devised to investigate the encoding of knowledge -- e.g. syntactic structure -- in contextual representations. Probes are often designed for simplicity, which has…
How (Non-)Optimal is the Lexicon?
Tiago Pimentel, Irene Nikkarinen, Kyle Mahowald +2
The mapping of lexical meanings to wordforms is a major feature of natural languages. While usage pressures might assign short words to frequent meanings (Zipf's law of abbreviatio…
Finding Concept-specific Biases in Form--Meaning Associations
Tiago Pimentel, Brian Roark, Søren Wichmann +2
This work presents an information-theoretic operationalisation of cross-linguistic non-arbitrariness. It is not a new idea that there are small, cross-linguistic associations betwe…
What About the Precedent: An Information-Theoretic Analysis of Common Law
Josef Valvoda, Tiago Pimentel, Niklas Stoehr +2
In common law, the outcome of a new case is determined mostly by precedent cases, rather than by existing statutes. However, how exactly does the precedent influence the outcome of…
Quantifying Gender Bias Towards Politicians in Cross-Lingual Language Models
Karolina Stańczak, Sagnik Ray Choudhury, Tiago Pimentel +2
Recent research has demonstrated that large pre-trained language models reflect societal biases expressed in natural language. The present paper introduces a simple method for prob…