6 papers
WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans
Jun Zhang, Qiao Zhao, Cheng Cui +6
While the top model on OmniDocBench now reaches 96.34% overall on printed-document parsing, the ability of current models to handle challenging handwritten documents remains largel…
BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requests
Zetong Xiong, Qiao Zhao, Jun Zhang +20
Coding-agent benchmarks increasingly cover long-horizon, end-to-end, and interactive development, but typically retain one requested outcome or a fixed change sequence. Sequential…
SWE-Future: Forecast-Conditioned Data Synthesis for Future-Oriented Software Engineering Agents
Qiao Zhao, JianYing Qu, Jun Zhang +3
Realistic coding-agent benchmarks often replay public GitHub issues and pull requests, making them vulnerable to overlap with model pretraining, fine-tuning, synthetic-data generat…
Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA
Jun Zhang, JianYing Qu, Hanwen Du +3
We present Code-QA-Bench, a fully automated framework for synthesizing repository-level code understanding benchmarks that separates genuine code comprehension from documentation r…
Beyond Perfect APIs: A Comprehensive Evaluation of LLM Agents Under Real-World API Complexity
Doyoung Kim, Zhiwei Ren, Jie Hao +11
We introduce WildAGTEval, a benchmark designed to evaluate large language model (LLM) agents' function-calling capabilities under realistic API complexity. Unlike prior work that a…
STOC-TOT: Stochastic Tree-of-Thought with Constrained Decoding for Complex Reasoning in Multi-Hop Question Answering
Zhenyu Bi, Daniel Hajialigol, Zhongkai Sun +2
Multi-hop question answering (MHQA) requires a model to retrieve and integrate information from multiple passages to answer a complex question. Recent systems leverage the power of…