7 papers
What Does an LLM-Agent Leaderboard Rank Actually Compare?
Wei-Jung Huang
An LLM-agent leaderboard invites a familiar inference: an agent ranked above another is the better agent. Public evaluation logs may not support that conclusion when systems differ…
Can Training Logs Make Model Comparisons More Precise?
Wei-Jung Huang
Comparing stochastically trained models requires estimating both a performance difference and its uncertainty from repeated runs. We study whether training logs from those same run…
ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision
Wei-Jung Huang, Bonan Shen
LLM-agent evaluations often produce task outcomes long before the full benchmark run is complete. A partial score is tempting to report, but it does not show whether the observed t…
How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks
Wei-Jung Huang
Agent benchmarks often compare two agents after all tasks have run, but costly evaluations make partial runs tempting. A task fraction alone does not show whether a partial run sup…
Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows
Wei-Jung Huang
Product data scientists often ask LLM-based agents to help with recurring execution tasks such as cleaning data, writing SQL, choosing statistical tests, and formatting results. Re…
AdaStop: Cost-Aware Early Stopping for DNN Test Selection
Bonan Shen, Wei-Jung Huang, Xin Liu +2
Existing methods for testing deep neural networks (DNNs) primarily prioritize test inputs likely to reveal model faults under a fixed labeling budget. In practice, choosing that bu…