17 papers
Predicting Privileged Information Effectiveness for On-Policy Self-Distillation
Kaijian Zou, Emiliano Penaloza, MohammadHossein Rezaei +5
On-policy self-distillation (OPSD) enables large language models (LLMs) to learn from privileged information (PI) without a separate teacher model. However, prior work largely trea…
Evidence-First Agentic Verification
Reza Sanayei, Razvan-Gabriel Dumitru, Sunjie Hou +7
Long-horizon, semi-verifiable agentic tasks often rely on rubric-based evaluation, but the evidence needed to assess these tasks can be spread across the environment state and the…
Structural properties of admissible subgraphs and the depth of symbolic powers of cover ideals
Nguyen Thu Hang, Nguyen Thi Thanh Tam, Thanh Vu
Let be a simple graph with cover ideal in a polynomial ring $S = \k[x_1, \ldots, x_n]$. For each integer , let denote the maximum size of an ordered ma…
CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data
Vipul Gupta, Zihao Wang, Razvan-Gabriel Dumitru +3
Evaluations should do more than measure a models current performance. They should tell us what to fix for the next model iteration and provide a way to generate targeted post train…
Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers
MohammadHossein Rezaei, Anas Mahmoud, Zihao Wang +6
Rubrics have emerged as an alternative to RLVR in open-ended domains where a single ground-truth final answer is not available. Existing rubric-based training methods rely on an LL…
From Residuals to Reasons: LLM-Guided Mechanism Inference from Tabular Data
Mohammad R. Rezaei, Rahul G. Krishnan
A persistent challenge in machine learning for scientific applications is jointly achieving prediction and understanding. Statistical models excel on structured data but operate as…
Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei +5
Reinforcement learning with verifiable rewards has made post-training highly effective when correctness can be checked automatically. However, many important model behaviors requir…
Reward Hacking in Rubric-Based Reinforcement Learning
Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang +3
Reinforcement learning with verifiable rewards has enabled strong post-training gains in domains such as math and coding, though many open-ended settings rely on rubric-based rewar…
SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
Mohit Raghavendra, Soham Dan, Miguel Romero Calvo +12
We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refac…
Commonsense Knowledge with Negation: A Resource to Enhance Negation Understanding
Zijie Wang, MohammadHossein Rezaei, Farzana Rashid +1
Negation is a common and important semantic feature in natural language, yet Large Language Models (LLMs) struggle when negation is involved in natural language understanding tasks…
Process Rewards for Outcome-Guided Reasoning Steps in LLM Reinforcement Learning
Mohammad Rezaei, Jens Lehmann, Sahar Vahdati
Mathematical reasoning in large language models has improved substantially with reinforcement learning using verifiable rewards, where final answers can be checked automatically an…
MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers
Chaithanya Bandi, Razvan-Gabriel Dumitru, Ben Hertzberg +20
The Model Context Protocol (MCP) is emerging as a standard interface through which large language model (LLM) agents discover and invoke external tools. However, existing MCP evalu…
Online Rubrics Elicitation from Pairwise Comparisons
MohammadHossein Rezaei, Robert Vacareanu, Zihao Wang +4
Rubrics provide a flexible way to train LLMs on open-ended long-form answers where verifiable rewards are not applicable and human preferences provide coarse signals. Prior work sh…
Making Language Models Robust Against Negation
MohammadHossein Rezaei, Eduardo Blanco
Negation has been a long-standing challenge for language models. Previous studies have shown that they struggle with negation in many natural language understanding tasks. In this…
EgoNormia: Benchmarking Physical Social Norm Understanding
MohammadHossein Rezaei, Yicheng Fu, Phil Cuvin +4
Human activity is moderated by norms; however, supervision for normative reasoning is sparse, particularly where norms are physically- or socially-grounded. We thus present EGONORM…
Paraphrasing in Affirmative Terms Improves Negation Understanding
MohammadHossein Rezaei, Eduardo Blanco
Negation is a common linguistic phenomenon. Yet language models face challenges with negation in many natural language understanding tasks such as question answering and natural la…
Interpreting Indirect Answers to Yes-No Questions in Multiple Languages
Zijie Wang, Md Mosharaf Hossain, Shivam Mathur +8
Yes-no questions expect a yes or no for an answer, but people often skip polar keywords. Instead, they answer with long explanations that must be interpreted. In this paper, we foc…