2 citations · 3 across the 27 of their papers we have counts for
9 papers · 1 filter
How Language Models Choose Sides: Internal Representations of Instruction Hierarchy
Enrique Balp-Straffon, Chih-Hao Hsu, Rushiraj Gadhvi +3
We study how instruction-tuned LLMs arbitrate direct conflicts between system and user instructions. We introduce a benchmark of 41 paired constraints with deterministic verifiers…
Seven simple steps for log analysis in AI systems
Magda Dubois, Ekin Zorer, Maia Hamin +17
AI systems produce large volumes of logs as they interact with tools and users. Analysing these logs can help understand model capabilities, propensities, and behaviours, or assess…
Judge Reliability Harness: Stress Testing the Reliability of LLM Judges
Sunishchal Dev, Andrew Sloan, Joshua Kavner +2
We present the Judge Reliability Harness, an open source library for constructing validation suites that test the reliability of LLM judges. As LLM based scoring is widely deployed…
ProMoral-Bench: Evaluating Prompting Strategies for Moral Reasoning and Safety in LLMs
Rohan Subramanian Thomas, Shikhar Shiromani, Abdullah Chaudhry +4
Prompt design significantly impacts the moral competence and safety alignment of large language models (LLMs), yet empirical comparisons remain fragmented across datasets and model…
Emergent Persuasion: Will LLMs Persuade Without Being Prompted?
Vincent Chang, Thee Ho, Sunishchal Dev +4
With the wide-scale adoption of conversational AI systems, AI are now able to exert unprecedented influence on human opinion and beliefs. Recent work has shown that many Large Lang…
AgentChangeBench: A Multi-Dimensional Evaluation Framework for Goal-Shift Robustness in Conversational AI
Manik Rana, Calissa Man, Anotida Expected Msiiwa +5
Goal changes are a defining feature of real world multi-turn interactions, yet current agent benchmarks primarily evaluate static objectives or one-shot tool use. We introduce Agen…
Limits of Emergent Reasoning of Large Language Models in Agentic Frameworks for Deterministic Games
Chris Su, Harrison Li, Matheus Marques +3
Recent work reports that Large Reasoning Models (LRMs) undergo a collapse in performance on solving puzzles beyond certain perplexity thresholds. In subsequent discourse, questions…
LIFT: Likelihood-Based Intervention Faithfulness Training
Anand Swaroop, Akshat Nallani, Saksham Uboweja +7
Chain-of-thought (CoT) reasoning improves LLM accuracy on complex tasks, but CoTs are often unfaithful; that is, the model does not rely on the stated reasoning steps to produce it…
Recommendations and Reporting Checklist for Rigorous & Transparent Human Baselines in Model Evaluations
Kevin L. Wei, Patricia Paskov, Sunishchal Dev +6
In this position paper, we argue that human baselines in foundation model evaluations must be more rigorous and more transparent to enable meaningful comparisons of human vs. AI pe…