#agent evaluation
10 papers match
Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Game
Niklas Bauer, Lars Benedikt Kaesberg, Akiko Aizawa +3
The paper introduces ParliamentBench, an open-source benchmark based on the Secret Hitler game, to evaluate large language model agents on deception, persuasion, and reasoning unde…
Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation
Stefan Krsteski, Charlotte Meyer, Guillaume Allegre +2
The paper introduces Messier, a unified corpus of 957,253 standardized evaluation records spanning thousands of agents, tasks, and benchmarks, to enable cross‑benchmark analysis an…
AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
Kai Chen, Zichen Ding, Jiaye Ge +22
The paper presents AgentCompass, an open‑source infrastructure that standardizes and simplifies the evaluation of large‑language‑model based autonomous agents by separating benchma…
Copy-on-Write Scoring: Application-Specific Agent Evaluations
Joanna Roy, Sven Hoelzel
The paper presents Copy-on-Write (CoW) Scoring, a framework that uses PostgreSQL's copy‑on‑write mechanism to isolate and evaluate LLM‑based agents' database write operations withi…
JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks
Lanbo Lin, Jiayao Liu, Tianyuan Yang +5
JADE is a two‑layer evaluation system that encodes expert knowledge as predefined evaluation skills and adds claim‑level, evidence‑gated assessment to more reliably evaluate AI age…
Rethinking the Evaluation of Harness Evolution for Agents
Yike Wang, Huaisheng Zhu, Zhengyu Hu +7
The paper reexamines how automatic harness evolution for large language model agents is evaluated, comparing it to simple test‑time scaling baselines and finding that it offers lim…