8 papers
SchemeArena: Factorized Stress Testing of Scheming in LLM Agents
Jie Ruan, Inderjeet Nair, Amy Liu +3
We study scheming in LLM agents, in which agents covertly pursue misaligned goals. Our focus is to understand how scheming arises from the interaction of key factors, such as instr…
Value-Conflict Diagnostics Reveal Widespread Alignment Faking in Language Models
Inderjeet Nair, Jie Ruan, Lu Wang
Alignment faking, where a model behaves aligned with developer policy when monitored but reverts to its own preferences when unobserved, is a concerning yet poorly understood pheno…
Context Over Content: Exposing Evaluation Faking in Automated Judges
Manan Gupta, Inderjeet Nair, Lu Wang +1
The paradigm has become the operational backbone of automated AI evaluation pipelines, yet rests on an unverified assumption: that judges evaluate text st…
Do Language Models Think Consistently? A Study of Value Preferences Across Varying Response Lengths
Inderjeet Nair, Lu Wang
Evaluations of LLMs' ethical risks and value inclinations often rely on short-form surveys and psychometric tests, yet real-world use involves long-form, open-ended responses -- le…
ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists
Jie Ruan, Inderjeet Nair, Shuyang Cao +14
This paper introduces ExpertLongBench, an expert-level benchmark containing 11 tasks from 9 domains that reflect realistic expert workflows and applications. Beyond question answer…
GenSco: Can Question Decomposition based Passage Alignment improve Question Answering?
Barah Fazili, Koustava Goswami, Natwar Modani +1
Retrieval augmented generation (RAG) with large language models (LLMs) for Question Answering (QA) entails furnishing relevant context within the prompt to facilitate the LLM in an…
MIDGARD: Self-Consistency Using Minimum Description Length for Structured Commonsense Reasoning
Inderjeet Nair, Lu Wang
We study the task of conducting structured reasoning as generating a reasoning graph from natural language input using large language models (LLMs). Previous approaches have explor…
Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering
Inderjeet Nair, Shwetha Somasundaram, Apoorv Saxena +1
We address the task of evidence retrieval for long document question answering, which involves locating relevant paragraphs within a document to answer a question. We aim to assess…