7 papers
Different Facets of Verbalised Overconfidence: an Interpretability Study
Davide Mazzaccara, Leonardo Bertolazzi, Raffaella Bernardi
Large language models tend to overconfidence, giving assertive answers when the evidence suggests hedging or abstention. Using controlled reasoning scenarios that manipulate logica…
FALSIFYBENCH: Evaluating Inductive Reasoning in LLMs with Rule Discovery Games
Leonardo Bertolazzi, Katya Tentori, Raffaella Bernardi
Large language models (LLMs) are increasingly deployed as autonomous agents in scientific tasks. Yet whether these systems can effectively engage in forms of inductive reasoning re…
How Language Models Conflate Logical Validity with Plausibility: A Representational Analysis of Content Effects
Leonardo Bertolazzi, Sandro Pezzelle, Raffaella Bernardi
Both humans and large language models (LLMs) exhibit content effects: biases in which the plausibility of the semantic content of a reasoning problem influences judgments regarding…
Teaching Small Language Models to Learn Logic through Meta-Learning
Leonardo Bertolazzi, Manuel Vargas Guzmán, Raffaella Bernardi +2
Large language models (LLMs) are increasingly evaluated on reasoning tasks, yet their logical abilities remain contested. To address this, we study LLMs' reasoning in a well-define…
The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate It
Leonardo Bertolazzi, Philipp Mondorf, Barbara Plank +1
The ability of large language models (LLMs) to validate their output and identify potential errors is crucial for ensuring robustness and reliability. However, current research ind…
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi +17
There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments, raising questions about the validity of these evaluations, as well as their reprodu…