Statistical Evaluation of Anomaly Detectors for Sequences

Abstract

Although precision and recall are standard performance measures for anomaly detection, their statistical properties in sequential detection settings are poorly understood. In this work, we formalize a notion of precision and recall with temporal tolerance for point-based anomaly detection in sequential data. These measures are based on time-tolerant confusion matrices that may be used to compute time-tolerant variants of many other standard measures. However, care has to be taken to preserve interpretability. We perform a statistical simulation study to demonstrate that precision and recall may overestimate the performance of a detector, when computed with temporal tolerance. To alleviate this problem, we show how to obtain null distributions for the two measures to assess the statistical significance of reported results.

5 pages, 6 figures, accepted at the 6th KDD Workshop on Mining and Learning from Time Series (KDD MiLeTS 2020), source code available at https://github.com/diozaka/anomaly-eval

References in corpus (5)

• Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
• Time-Series Anomaly Detection Service at Microsoft
• Event coincidence analysis for quantifying statistical interrelationships between event time series: on the role of flood events as possible triggers of epidemic outbreaks
• Precision and Recall for Time Series
• Does Terrorism Trigger Online Hate Speech? On the Association of Events and Time Series

Cited by in corpus (2)

• SoftED: Metrics for Soft Evaluation of Time Series Event Detection
• Data-driven Design of Context-aware Monitors for Hazard Prediction in Artificial Pancreas Systems