6 papers
ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders
Zhongyuan Peng, Dan Huang, Chuyu Zhang +8
The recent emergence of vibe-coding workflows is changing what coding agents are expected to do. Instead of merely completing code under fully specified instructions, agents are in…
DLEBench: Evaluating Small-scale Object Editing Ability for Instruction-based Image Editing Model
Shibo Hong, Boxian Ai, Jun Kuang +5
Significant progress has been made in the field of Instruction-based Image Editing Models (IIEMs). However, while these models demonstrate plausible adherence to instructions and s…
CoDiQ: Test-Time Scaling for Controllable Difficult Question Generation
Zhongyuan Peng, Caijun Xu, Changyi Xiao +4
Large Reasoning Models (LRMs) benefit substantially from training on challenging competition-level questions. However, existing automated question synthesis methods lack precise di…
FRABench and UFEval: Unified Fine-grained Evaluation with Task and Aspect Generalization
Shibo Hong, Jiahao Ying, Haiyuan Liang +4
Evaluating open-ended outputs of Multimodal Large Language Models has become a bottleneck as model capabilities, task diversity, and modality rapidly expand. Existing ``MLLM-as-a-J…
Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang +9
Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human values. However, noisy preferences in human feedback can lead to reward misgeneralizatio…
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks
Yixin Cao, Shibo Hong, Xinze Li +24
Large Language Models (LLMs) are advancing at an amazing speed and have become indispensable across academia, industry, and daily applications. To keep pace with the status quo, th…