ORCA-bench: How Ready Are Language Model Agents for Oncall?

Albert Gong, Kyuseong Choi, Abhineet Agarwal +5

The paper presents ORCA-bench, a benchmark that evaluates large language model agents on on-call root cause analysis tasks using real telemetry data from a live microservice system…