Local Evaluation of Time Series Anomaly Detection Algorithms

Abstract

In recent years, specific evaluation metrics for time series anomaly detection algorithms have been developed to handle the limitations of the classical precision and recall. However, such metrics are heuristically built as an aggregate of multiple desirable aspects, introduce parameters and wipe out the interpretability of the output. In this article, we first highlight the limitations of the classical precision/recall, as well as the main issues of the recent event-based metrics -- for instance, we show that an adversary algorithm can reach high precision and recall on almost any dataset under weak assumption. To cope with the above problems, we propose a theoretically grounded, robust, parameter-free and interpretable extension to precision/recall metrics, based on the concept of ``affiliation'' between the ground truth and the prediction sets. Our metrics leverage measures of duration between ground truth and predictions, and have thus an intuitive interpretation. By further comparison against random sampling, we obtain a normalized precision/recall, quantifying how much a given set of results is better than a random baseline prediction. By construction, our approach keeps the evaluation local regarding ground truth events, enabling fine-grained visualization and interpretation of algorithmic results. We compare our proposal against various public time series anomaly detection datasets, algorithms and metrics. We further derive theoretical properties of the affiliation metrics that give explicit expectations about their behavior and ensure robustness against adversary strategies.

arXiv commentAccepted by KDD 2022 Research Track

References in corpus (6)

• Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding
• Selective review of offline change point detection methods
• Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications
• Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmark
• An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series
• Novel semi-metrics for multivariate change point analysis and anomaly detection

Cited by in corpus (8)

• Deep Learning for Time Series Anomaly Detection: A Survey
• DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detection
• Online Model-based Anomaly Detection in Multivariate Time Series: Taxonomy, Survey, Research Challenges and Future Directions
• DACAD: Domain Adaptation Contrastive Learning for Anomaly Detection in Multivariate Time Series
• A Comparison of Deep Learning Architectures for Spacecraft Anomaly Detection
• SoK: Evaluations in Industrial Intrusion Detection Research
• TriP-LLM: A Tri-Branch Patch-wise Large Language Model Framework for Time-Series Anomaly Detection
• Noise-Resilient Point-wise Anomaly Detection in Time Series Using Weak Segment Labels