4 papers
From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agents
Xingyu Su, Abhishek Kumar, Qing Ping +5
On-policy self-distillation (OPSD) has become a popular recipe for post-training LLM agents. It supervises the agent model at the token level with a stronger teacher view of the sa…
PERSONAJUDGE: Simulating Individual Human Preference Judgments with Evaluator-Specific Demonstration Data
Zeyu He, Xuan Qi, Subramanian Chidambaram +4
Large language models increasingly serve as judges in AI evaluation, but current approaches rely on consensus preferences that ignore individual evaluator variation. We propose a n…
An Empirical Study of Automating Agent Evaluation
Kang Zhou, Sangmin Woo, Haibo Ding +15
Agent evaluation requires assessing complex multi-step behaviors involving tool use and intermediate reasoning, making it costly and expertise-intensive. A natural question arises:…
PILOT: Steering Synthetic Data Generation with Psychological & Linguistic Output Targeting
Caitlin Cisar, Emily Sheffield, Joshua Drake +5
Generative AI applications commonly leverage user personas as a steering mechanism for synthetic data generation, but reliance on natural language representations forces models to…