Log analysis is necessary for credible evaluation of AI agents

Peter Kirgis, Sayash Kapoor, Stephan Rabanser +8

Agent benchmarks typically report only final outcomes: pass or fail. This threatens evaluation credibility in three ways. First, scores may be inflated or deflated by shortcuts and…