Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents

Noam Koren, Roy Bar-Haim, Abigail Goldsteen

Task-oriented conversational agents are evaluated using curated or automatically generated benchmarks, yet benchmark quality is rarely assessed. Poor benchmarks may contain inconsi…