12 papers · 1 filter
Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?
Yangda Peng, Yunjia Qi, Hao Peng +11
Rubric-based scoring has become a widely used paradigm in model evaluation, typically with LLM-as-a-Judge (LaaJ) for rubric scoring. However, the reliability of LaaJ for rubric sco…
StoryAlign: Evaluating and Training Reward Models for Story Generation
Haotian Xia, Hao Peng, Yunjia Qi +4
Story generation aims to automatically produce coherent, structured, and engaging narratives. Although large language models (LLMs) have significantly advanced text generation, sto…
WildReward: Learning Reward Models from In-the-Wild Human Interactions
Hao Peng, Yunjia Qi, Xiaozhi Wang +3
Reward models (RMs) are crucial for the training of large language models (LLMs), yet they typically rely on large-scale human-annotated preference pairs. With the widespread deplo…
On the Paradoxical Interference between Instruction-Following and Task Solving
Yunjia Qi, Hao Peng, Xintong Shi +5
Instruction following aims to align Large Language Models (LLMs) with human intent by specifying explicit constraints on how tasks should be performed. However, we reveal a counter…
StoryWriter: A Multi-Agent Framework for Long Story Generation
Haotian Xia, Hao Peng, Yunjia Qi +4
Long story generation remains a challenge for existing large language models (LLMs), primarily due to two main factors: (1) discourse coherence, which requires plot consistency, lo…
VerIF: Verification Engineering for Reinforcement Learning in Instruction Following
Hao Peng, Yunjia Qi, Xiaozhi Wang +3
Reinforcement learning with verifiable rewards (RLVR) has become a key technique for enhancing large language models (LLMs), with verification engineering playing a central role. H…