Attack Success Rate Is Not a Number: On Measurement Validity in Agentic AI Security Evaluation

Chetan Pathade, Prathamesh Pawar, Shubham Patil

Attack success rate (ASR) is the headline metric in nearly every published evaluation of attacks on, and defenses for, LLM agents. We argue that ASR as currently used is not a sing…