5 papers
Automated Benchmark Auditing for AI Agents and Large Language Models
Junlin Wang, Federico Bianchi, Shang Zhu +4
Modern AI benchmarks operate at a complexity that outpaces traditional verification methods. Tasks authored by domain experts often contain implicit assumptions, incomplete environ…
Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning
Qin Zhang, Peiyu Jing, Hong-Xing Yu +7
Video generation models are increasingly used as world simulators for storytelling, simulation, and embodied AI. As these models advance, a key question arises: do generated videos…
EvoLM: In Search of Lost Language Model Training Dynamics
Zhenting Qi, Fan Nie, Alexandre Alahi +6
Modern language model (LM) training has been divided into multiple stages, making it difficult for downstream developers to evaluate the impact of design choices made at each stage…
When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought
Yiyang Zhou, Haoqin Tu, Zijun Wang +11
We propose MIRA, a new benchmark designed to evaluate models in scenarios where generating intermediate visual images is essential for successful reasoning. Unlike traditional CoT…
UQ: Assessing Language Models on Unsolved Questions
Fan Nie, Ken Ziyu Liu, Zihao Wang +11
Benchmarks shape progress in AI research. A useful benchmark should be both difficult and realistic: questions should challenge frontier models while also reflecting real-world usa…