21 papers
Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation
Jon-Paul Cacioli
We adapted the Reliable Change Index (RCI; Jacobson and Truax, 1991) from clinical psychology to item-level LLM version comparison on 2,000 MMLU-Pro items (K=10 samples at T=0.7).…
Instruction Complexity Induces Positional Collapse in Adversarial LLM Evaluation
Jon-Paul Cacioli
When instructed to underperform on multiple-choice evaluations, do language models engage with question content or fall back on positional shortcuts? We map the boundary between th…
Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging
Jon-Paul Cacioli
A predecessor pilot (Cacioli, 2026) found that Llama-3-8B implements prompted sandbagging as positional collapse rather than answer avoidance. However, fixed option ordering in MML…
Below-Chance Blindness: Prompted Underperformance in Small LLMs Produces Positional Bias Rather than Answer Avoidance
Jon-Paul Cacioli
Detecting sandbagging--the deliberate underperformance on capability evaluations--is an open problem in AI safety. We tested whether symptom validity testing (SVT) logic from clini…
Distilling Self-Consistency into Verbal Confidence: A Pre-Registered Negative Result and Post-Hoc Rescue on Gemma 3 4B
Jon-Paul Cacioli
Small instruct-tuned LLMs produce degenerate verbal confidence under minimal elicitation: ceiling rates above 95%, near-chance Type-2 AUROC, and Invalid validity profiles. We test…
Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs: A Pre-Registered Psychometric Validity Screen
Jon-Paul Cacioli
Verbal confidence elicitation is widely used to extract uncertainty estimates from LLMs. We tested whether seven instruction-tuned open-weight models (3-9B parameters, four familie…
Categorical Perception in Large Language Model Hidden States: Structural Warping at Digit-Count Boundaries
Jon-Paul Cacioli
Categorical perception (CP) -- enhanced discriminability at category boundaries -- is among the most studied phenomena in perceptual psychology. This paper reports that analogous g…
Cross-Entropy Is Load-Bearing: A Pre-Registered Scope Test of the K-Way Energy Probe on Bidirectional Predictive Coding
Jon-Paul Cacioli
Cacioli (2026) showed that the K-way energy probe on standard discriminative predictive coding networks reduces approximately to a monotone function of the log-softmax margin. The…
Domain-level metacognitive monitoring in frontier LLMs: A 33-model atlas
Jon-Paul Cacioli
Aggregate metacognitive quality scores mask within-model variation across MMLU benchmark domains. We administered 1,500 MMLU items (250 per domain, under an a priori six-domain gro…
The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring
Jon-Paul Cacioli
We introduce a cross-domain behavioural assay of monitoring-control coupling in LLMs, grounded in the Nelson and Narens (1990) metacognitive framework and applying human psychometr…
Concurrent Criterion Validation of a Validity Screen for LLM Confidence Signals via Selective Prediction
Jon-Paul Cacioli
The validity screen (Cacioli, 2026d, 2026e) classifies LLM confidence signals as Valid, Indeterminate, or Invalid. We test whether these classifications predict selective predictio…
Screen Before You Interpret: A Portable Validity Protocol for Benchmark-Based LLM Confidence Signals
Jon-Paul Cacioli
LLM confidence signals are used for abstention, routing, and safety-critical decisions. No standard practice exists for checking whether a confidence signal carries item-level info…
Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report
Jon-Paul Cacioli
Clinical personality assessment screens response validity before interpreting substantive scales. LLM evaluation does not. We apply the validity scaling framework from the PAI and…
K-Way Energy Probes for Metacognition Reduce to Softmax in Discriminative Predictive Coding Networks
Jon-Paul Cacioli
We present this as a negative result with an explanatory mechanism, not as a formal upper bound. Predictive coding networks (PCNs) admit a K-way energy probe in which each candidat…
Quantisation Reshapes the Metacognitive Geometry of Language Models
Jon-Paul Cacioli
We report that model quantisation restructures domain-level metacognitive efficiency in LLMs rather than degrading it uniformly. Evaluating Llama-3-8B-Instruct on the same 3,000 qu…
Exemplar Retrieval Without Overhypothesis Induction: Limits of Distributional Sequence Learning in Early Word Learning
Jon-Paul Cacioli
Background: Children do not simply learn that balls are round and blocks are square. They learn that shape is the kind of feature that tends to define object categories -- a second…
Same Geometry, Opposite Noise: Transformer Magnitude Representations Lack Scalar Variability
Jon-Paul Cacioli
Scalar variability -- the finding that representational noise scales proportionally with magnitude, producing a constant coefficient of variation -- is a hallmark of biological mag…
Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models
Jon-Paul Cacioli
How do transformer language models represent magnitude? Recent work disagrees: some find logarithmic spacing, others linear encoding, others per-digit circular representations. We…
LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy
Jon-Paul Cacioli
Large language models (LLMs) are evaluated for calibration using metrics such as Expected Calibration Error that conflate two distinct components: the model's ability to discrimina…
Repetition Without Exclusivity: Scale Sensitivity of Referential Mechanisms in Child-Scale Language Models
Jon-Paul Cacioli
We present the first systematic evaluation of mutual exclusivity (ME) -- the bias to map novel words to novel referents -- in text-only language models trained on child-directed sp…
Do LLMs Know What They Know? Measuring Metacognitive Efficiency with Signal Detection Theory
Jon-Paul Cacioli
Standard evaluation of LLM confidence relies on calibration metrics (ECE, Brier score) that conflate two capacities: how much a model knows (Type-1 accuracy) and how well its confi…