Statistical Evaluation of Anomaly Detectors for Sequences
arXiv:2008.05788
Abstract
Although precision and recall are standard performance measures for anomaly detection, their statistical properties in sequential detection settings are poorly understood. In this work, we formalize a notion of precision and recall with temporal tolerance for point-based anomaly detection in sequential data. These measures are based on time-tolerant confusion matrices that may be used to compute time-tolerant variants of many other standard measures. However, care has to be taken to preserve interpretability. We perform a statistical simulation study to demonstrate that precision and recall may overestimate the performance of a detector, when computed with temporal tolerance. To alleviate this problem, we show how to obtain null distributions for the two measures to assess the statistical significance of reported results.
5 pages, 6 figures, accepted at the 6th KDD Workshop on Mining and Learning from Time Series (KDD MiLeTS 2020), source code available at https://github.com/diozaka/anomaly-eval