#benchmark evaluation
32 resultsAgenticASR: Refining Speech Recognition in Real-World Scenarios via an Agentic Approach
Zixuan Jiang, Binghao Qiang, Jiaying Chi +3
The paper introduces AgenticASR, an architecture that continuously refines speech recognition output to remove disfluencies and preserve speaker intent during live audio streams.
Albilich: Steerable Proof-State Orchestration for LLM-Based Mathematical Research with CAS Integration
Ting Gong, Michael Ruofan Zeng, Yong Yang
Albilich is an open‑source agentic framework that lets large language models conduct long‑horizon mathematical research by integrating computer algebra systems, literature retrieva…
When Specifications Conflict: A Symmetry-Based Framework for Measuring LLM Preferences
Tairan Wang, Liang Zhou, Zikang Zhan +1
The paper presents a symmetry‑based experimental framework for measuring how large language models resolve conflicts between competing specifications, and evaluates systematic pref…
Rethinking Inference-Time Scaling in Local Computer-Use Agents: Failure Modes and Compute Tradeoffs
Woongkyu Lee, Jungwook Choi
The paper empirically studies how different inference-time scaling strategies affect the performance and failure modes of locally deployed autonomous computer-use agents under hard…
How Benchmarks Mis-Score Computer-Use Agents
Zihan Dong, Zhiyuan Ma, Zekun Wang +5
The paper examines how current benchmarks for computer-use agents often give inaccurate scores due to issues in task design, trajectory observation, scoring, and reporting, and pro…
Evidence-Ledger Adjudication for Claim-Evidence Traceability
Gengyu Chen, Yongjie Yu, Weiling Wang
The paper introduces evidence-ledger adjudication, a workflow that pairs AI‑generated claims with evidence packets, assigns support relations, and routes unsupported or contradicto…
See2Think: Do Multimodal Models Really Use Intermediate Visual States?
Siyu Yan, Zhuoran Yan, Haiying Xu +10
The paper presents See2Think, an evaluation framework and benchmark for testing whether multimodal large language models actually use intermediate visual states during reasoning, a…
MemSecBench: Tracking Agent Memory Poisoning from Persistence to Consequence and Repair
Xuanze Chen, Xukang Xie, Wentao Fu +3
The paper presents MemSecBench, a benchmark that evaluates how malicious instructions can persist, be executed, and be repaired in agent memory systems across different memory and…
MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis
Yihao Chen, Shi Chang, Khaled Chawa +4
MindForge automatically turns open‑source command‑line programs into source‑free environments that expose only compiled executables and documentation, enabling the training of smal…
Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks
Jeff Mohl, Nelson Gardner-Challis, Magda Dubois +6
The paper presents automated AI scanners that analyze benchmark transcripts to detect validity flaws such as ground‑truth leakage, tool failures, guessing vulnerabilities, and ambi…
Visual Credit Audit for Multimodal Spatial Reasoning
Feixiang Liu, Qiang Qiu, Lanbo Sun +3
The paper introduces Visual Credit Audit (VCA), a method to quantify how much an image actually contributes to a multimodal model’s answer on spatial reasoning tasks, separating co…
When benchmark inferences do not compose: Projectibility in AI evaluation
Brett Reynolds
The paper examines how AI benchmark results are extrapolated to broader claims, introducing a non‑composition principle that warns against automatically chaining supported inferenc…
Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
Jiaang Li, Chengzu Li, Zhaochong An +4
The paper investigates why multimodal large language models often ignore visual evidence, using image reconstruction and a new benchmark (WhatIfVis) to measure how well models bala…
Do Code Language Models Use Tests? A Behavioral and Representational Study of Test-Driven Code Generation
Yunhao Liang, Chengguang Gan, Ruixuan Ying +3
The paper investigates how code language models respond to test cases in prompts, analyzing whether tests act as executable specifications or merely extra context, and finds that t…
The Tool Illusion: Rethinking Tool Use in Web Agents
Renze Lou, Baolin Peng, Wenlin Yao +5
The paper presents a large, controlled empirical study of tool use in web agents, examining how different tools, backbone models, and frameworks affect performance across multiple…
Can We Trust Item Response Theory for AI Evaluation?
Han Jiang, Sunbeom Kwon, Jinwen Luo +2
The paper investigates how well item response theory (IRT) works for evaluating large language model benchmarks, highlighting challenges when benchmark data differ from traditional…
Overlapping Network Community Detection Using Sparse Backbones
Zihe Zhou, Samin Aref
The paper introduces Highway, a scalable algorithm that detects overlapping communities in networks by leveraging a sparse backbone representation to improve both accuracy and effi…
The Joint Effect of Quantization and Sampling Temperature on LLM Safety Alignment: A Factorial Analysis
Hari Prasad, Ritam Pal
The paper investigates how model quantization and higher sampling temperatures jointly affect the safety alignment of instruction-tuned large language models, finding that quantiza…
Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0
Wenxiao Wang, Priyatham Kattakinda, Soheil Feizi
The paper evaluates whether gains from agent-optimization methods compound over successive optimization phases in a continual‑learning setting, using hard tasks from Terminal‑Bench…
What Do Temporal Graph Learning Models Learn?
Abigail J. Hayes, Tobias Schumacher, Markus Strohmaier
The paper investigates which structural and temporal properties of graphs are actually captured by state‑of‑the‑art temporal graph learning models, using systematic tests on synthe…
PRISM Edit: One Vector for All Temporal Answers
Chen Huang, Qi Zheng, Ruiqin Zheng +2
The paper proposes PRISM Edit, a method that updates large language models to handle changing temporal facts by learning a single representation that can be modulated for different…
What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation
Farrukh Rahman
The paper examines how temporal video QA benchmark scores conflate task difficulty with the model's reliance on positional encodings versus visual sequence order, and introduces a…
Image Matching Filtering and Refinement by Planes and Beyond
Fabio Bellavia, Zhenjun Zhao, Luca Morelli +1
The paper evaluates state‑of‑the‑art filtering and refinement techniques for image matching, introduces a new method that combines planar constraints with cross‑correlation, and sh…
Measurement Risk in Supervised Financial NLP: Rubric and Metric Sensitivity on JF-ICR
Sidi Chang, Peiying Zhu, Yuxiao Chen +1
The paper investigates how the wording of evaluation rubrics and the choice of metrics affect the reliability of supervised financial NLP benchmarks, using a Japanese implicit‑comm…