21 citations · 63 across the 37 of their papers we have counts for
11 papers · 1 filter
From Language Models over Tokens to Language Models over Characters
Tim Vieira, Ben LeBrun, Mario Giulianelli +5
Modern language models are internally -- and mathematically -- distributions over strings rather than strings, posing numerous challenges for programm…
Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation
Tianyu Liu, Jirui Qi, Paul He +3
Recent work suggests that large language models enhanced with retrieval-augmented generation are easily influenced by the order, in which the retrieved documents are presented to t…
Efficiently Computing Susceptibility to Context in Language Models
Tianyu Liu, Kevin Du, Mrinmaya Sachan +1
One strength of modern language models is their ability to incorporate information from a user-input context when answering queries. However, they are not equally sensitive to the…
On the Proper Treatment of Tokenization in Psycholinguistics
Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi +3
Language models are widely used in computational psycholinguistics to test theories that relate the negative log probability (the surprisal) of a region of interest (a substring of…
The Foundations of Tokenization: Statistical and Computational Concerns
Juan Luis Gastaldi, John Terilla, Luca Malagutti +3
Tokenization - the practice of converting strings of characters from an alphabet into sequences of tokens over a vocabulary - is a critical step in the NLP pipeline. The use of tok…
Variational Best-of-N Alignment
Afra Amini, Tim Vieira, Elliott Ash +1
Best-of-N (BoN) is a popular and effective algorithm for aligning language models to human preferences. The algorithm works as follows: at inference time, N samples are drawn from…
What Do Language Models Learn in Context? The Structured Task Hypothesis
Jiaoda Li, Yifan Hou, Mrinmaya Sachan +1
Large language models (LLMs) exhibit an intriguing ability to learn a novel task from in-context examples presented in a demonstration, termed in-context learning (ICL). Understand…
On Affine Homotopy between Language Encoders
Robin SM Chan, Reda Boumasmoud, Anej Svete +8
Pre-trained language encoders -- functions that represent text as vectors -- are an integral component of many NLP tasks. We tackle a natural question in language encoder analysis:…
A Transformer with Stack Attention
Jiaoda Li, Jennifer C. White, Mrinmaya Sachan +1
Natural languages are believed to be (mildly) context-sensitive. Despite underpinning remarkably capable large language models, transformers are unable to model many context-free l…
Direct Preference Optimization with an Offset
Afra Amini, Tim Vieira, Ryan Cotterell
Direct preference optimization (DPO) is a successful fine-tuning strategy for aligning large language models with human preferences without the need to train a reward model or empl…
Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?
Andreas Opedal, Alessandro Stolfo, Haruki Shirakami +5
There is increasing interest in employing large language models (LLMs) as cognitive models. For such purposes, it is central to understand which properties of human cognition are w…