8 papers
A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents
Raghu Arghal, Fade Chen, Niall Dalton +6
Understanding an agent's goals helps explain and predict its behaviour, yet there is no established methodology for reliably attributing goals to agentic systems. We propose a fram…
Surprisal Minimisation over Goal-directed Alternatives Predicts Production Choice in Dialogue
Tom Utting, Mario Giulianelli, Arabella Sinclair
We model utterance production as probabilistic cost-sensitive choice over contextual alternatives, using information-theoretic notions of cost. We distinguish between goal-directed…
Probing for Reading Times
Eleftheria Tsipidi, Samuel Kiegeland, Francesco Ignazio Re +4
Probing has shown that language model representations encode rich linguistic information, but it remains unclear whether they also capture cognitive signals about human processing.…
Skewed Score: A statistical framework to assess autograders
Magda Dubois, Harry Coppock, Mario Giulianelli +3
The evaluation of large language model (LLM) outputs is increasingly performed by other LLMs, a setup commonly known as "LLM-as-a-judge", or autograders. While autograders offer a…
Reasoning aligns language models to human cognition
Gonçalo Guiomar, Elia Torre, Pehuen Moure +4
Do language models make decisions under uncertainty like humans do, and what role does chain-of-thought (CoT) reasoning play in the underlying decision process? We introduce an act…
Structure-Conditional Minimum Bayes Risk Decoding
Bryan Eikema, Anna Rutkiewicz, Mario Giulianelli
Minimum Bayes Risk (MBR) decoding has seen renewed interest as an alternative to traditional generation strategies. While MBR has proven effective in machine translation, where the…