32 papers
HappyWorld-Bench
Zhiqi Bai, Junai Cai, Yixin Chen +33
Evaluating world models requires assessing both the quality of the worlds they generate and their consistency and responsiveness under exploration, interaction, and modification. W…
VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks
Liyang Fan, Yingcheng Shi, Yongbin Li +7
Coding agents operate on real repository coding tasks, and persistent memory systems promise to reuse experience across tasks. Yet existing evaluations do not show whether those sy…
RubricRM: Generative Reward Modeling via Dynamic Rubrics for Image Generation and Editing
Zijian Kan, Wei Wang, Long Luo +6
Reward models play an essential role in aligning visual generative models, yet most existing visual reward models use a single scalar score or rely on fixed criteria that cannot ad…
The Illusion of : Evaluating the Breakdown of Counterfactual Reasoning in LLMs
Yucheng Wang, Yuetian Du, Zhengyi Liu +8
Counterfactual reasoning requires models to reason beyond the observed world and explain how altered conditions propagate through downstream consequences. Existing benchmarks large…
Living-Harness Is an Interactive-Agent Evolver
Yuetian Du, Yucheng Wang, He Xu +9
Large language model (LLM) agents may recover from a failure within an episode or after a retry, yet the same execution failure can recur in later tasks because post-episode feedba…
Science Edge Evaluation: SEE the Missing Step Toward Real Scientific Discovery
Taolin Han, Yuchen Zhang, Jinghang Wang +22
Large language models (LLMs) are increasingly involved in scientific discovery, yet it remains unclear whether they can support complex real laboratory science. Here we introduce S…
Right Answer, Wrong Method: Shortcut Hacking Misleads the Evaluation of LLM Reasoning on Frontier Science Benchmarks
Xuan Ren, Weiqi Zhai, Tianle Pu +3
Scientific reasoning benchmarks typically evaluate large language models (LLMs) using final-answer accuracy. However, a correct answer does not necessarily demonstrate the reasonin…
Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
Xinke Tong, Xuanming Zhang, Tianyi Tang +10
Do Large Language Models (LLMs) possess genuine structural reasoning, or merely rely on surface-level pattern matching? The financial domain, demanding numerical precision and mult…
Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery
Tianyun Zhong, Wangyi Jiang, Wei Wang +15
Large language models (LLMs) excel at answering pre-specified questions, yet their ability to navigate the open-ended, pre-conclusion stage of discovery remains largely unmeasured.…
LA4VLA: Learning to Act without Seeing via Language-Action Pretraining
Tao Lin, Yuxin Du, Yiran Mao +13
Vision-Language-Action (VLA) models are commonly pretrained on robot demonstrations by jointly mapping visual observations and language instructions to actions. However, dense visu…
Discovering Millions of Interpretable Features with Sparse Autoencoders
XinYang He, Wei Wang, Bing Zhao +5
Sparse autoencoders (SAEs) have emerged as a powerful tool for decomposing superposed language model representations into sparse and interpretable features. However, training SAEs…
ICBCBench: An Industry Consortium Benchmark for Financial Deep Research
Weiya Li, Zhiwei Tang, Yizhou He +10
With the rapid advancement of Deep Research Agents in knowledge-intensive domains such as finance, establishing reliable and domain-aligned evaluation standards remains a critical…
EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learning
Guhong Chen, Yingcheng Shi, Yongbin Li +6
Autonomous LLM training is often framed as recipe search, which leaves the training harness largely static. This limitation sharpens in agentic RL, where shifting bottlenecks and s…
Socratic-SWE: Self-Evolving Coding Agents via Trace-Derived Agent Skills
Chuan Xiao, Zhengbo Jiao, Shaobo Wang +5
LLM-driven software engineering agents have become a central testbed for real-world language-model capability, yet their training remains limited by the availability of high-qualit…
The Sim-to-Real Gap of Foundation Model Agents: A Unified MDP Perspective
Xiaoou Liu, Tiejin Chen, Weibo Li +2
Foundation model agents are increasingly deployed for real-world decision-making, but suffer from the sim-to-real gap. While robotics and classical control have mature frameworks t…
FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning
Zeyu Wang, Jingye Xu, Xiaogang Li +7
Current multimodal benchmarks for scientific reasoning primarily evaluate local information extraction -- models recognize symbols and values and then perform textual inference. Th…
BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs
Ben Wang, Xiaogang Li, Ruochen Gao +6
Current multimodal models handle static image recognition well, but intuitive physical reasoning remains a weakness. Predicting how objects will move and interact from a single ima…
MAS-Algorithm: A Workflow for Solving Algorithmic Programming Problems with a Multi-Agent System
Yuliang Xu, Xiang Xu, Yao Wan +2
Algorithmic problem solving serves as a rigorous testbed for evaluating structured reasoning in AI coding systems, as it directly reflects a model's ability to perform structured r…
Architectural Design Decisions in AI Agent Harnesses
Hu Wei
AI agent systems increasingly rely on reusable non-LLM engineering infrastructure that packages tool mediation, context handling, delegation, safety control, and orchestration. Yet…
MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoning
Lingyan Wu, Xiang Zheng, Weiqi Zhai +5
Medical reasoning can reach a plausible answer while overlooking an unsafe or unsupported intermediate step. Process-Level Reward Models (PRMs) can verify such steps, yet existing…
SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration
Jialong Chen, Xander Xu, Hu Wei +2
Large language model (LLM)-powered agents have demonstrated strong capabilities in automating software engineering tasks such as static bug fixing. However, in the real world, the…
Unmasking Reasoning Processes: A Process-aware Benchmark for Evaluating Structural Mathematical Reasoning in LLMs
Xiang Zheng, Weiqi Zhai, Wei Wang +15
Recent large language models (LLMs) achieve near-saturation accuracy on many established mathematical reasoning benchmarks, raising concerns about their ability to diagnose genuine…
DeepVision-103K: A Visually Diverse, Broad-Coverage, and Verifiable Mathematical Dataset for Multimodal Reasoning
Haoxiang Sun, Lizhen Xu, Bing Zhao +5
Reinforcement Learning with Verifiable Rewards (RLVR) has been shown effective in enhancing the visual reflection and reasoning capabilities of Large Multimodal Models (LMMs). Howe…
Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoning
Zhengbo Jiao, Shaobo Wang, Zifan Zhang +4
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Multimodal Large Language Models (MLLMs), yet how visual evidence is…