10 papers
REALHOP: Rethinking Multi-Hop Reasoning Evaluation via Behavioral Auditing
Jiawen Tao, Xiaokun Yuan, Yaoming Li +4
Complex questions often require multi-hop reasoning that connects facts distributed across sources or distant regions of a long context through intermediate steps. Benchmarks commo…
ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds
Ming Zhang, Zhenghao Xiang, Peizhong Gao +17
Scientific discovery begins where known problems end. There, AI systems must engage in exploration: framing hypotheses, designing experiments, and iterating on the results. However…
Breaking the Environment Wall: A Unified Framework for Preparing and Evolving Agent-Native Environments
Yukai Wu, Yuanjing Yang, Le Zhou +8
Many real-world tasks (e.g., office workflows, scientific experimentation) require LLM agents to interact repeatedly with their environments for context-dependent operations. Howev…
WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective
Chenxu Liu, Zilu Zou, Peizhong Gao +9
Human evaluation provides a direct measure of the quality of LLM-generated web applications. However, fitting human judgments through automated evaluation remains challenging. Stat…
AhaBench: Do Agents Turn Experience into Reusable Insights? A Long-Horizon Benchmark for Continual Learning
Zerui Cheng, Jiawei Xu, Huacan Chai +3
Can language agents continually learn from experience, turning earlier interactions into reusable capabilities? AhaBench evaluates this ability through exploration after solved hid…
Hy-MultiTurn: A Six-Dimensional Benchmark for Deep Multi-Turn Dialogue Understanding
Eileen Ye, Jiawen Tao, Yaoming Li +7
Long-running multi-turn interactions with chatbots and agents are now common, and a correct response often depends on remembering earlier details, tracking later revisions, identif…
Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training
Jiawen Tao, Miao Peng, Yaoming Li +7
Synthetic textbook data has improved language model pre-training, but prior work largely treats the benefit as a property of generated content or local rewriting style. We study a…
E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios
Weihuang Zheng, Tianyuan Zou, Eileen Ye +5
Large Language Models (LLMs) are increasingly deployed as agents that interact with stateful environments over multiple steps: gathering hidden information, composing tool calls, a…
Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI
Jiaqi Shao, Hanck Chen, Wei Zhang +2
Agent benchmarks increasingly evaluate repository editing, web research, terminal use, and long-horizon interaction. Their scores support capability claims only when the evaluation…
HyperLogic: A Hard, Forward-Authored Chinese Logical Reasoning Benchmark with Execution-Derived Answers
Ming Zhang, Qiyuan Peng, Yinxi Wei +16
Existing logic benchmarks primarily measure models' ability to answer reasoning questions directly. Scalable benchmarks often generate text from formal structures, which makes answ…