Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmark

Abstract

Much of the world's data is streaming, time-series data, where anomalies give significant information in critical situations; examples abound in domains such as finance, IT, security, medical, and energy. Yet detecting anomalies in streaming data is a difficult task, requiring detectors to process data in real-time, not batches, and learn while simultaneously making predictions. There are no benchmarks to adequately test and score the efficacy of real-time anomaly detectors. Here we propose the Numenta Anomaly Benchmark (NAB), which attempts to provide a controlled and repeatable environment of open-source tools to test and measure anomaly detection algorithms on streaming data. The perfect detector would detect all anomalies as soon as possible, trigger no false alarms, work with real-world time-series data across a variety of domains, and automatically adapt to changing statistics. Rewarding these characteristics is formalized in NAB, using a scoring algorithm designed for streaming data. NAB evaluates detectors on a benchmark dataset with labeled, real-world time-series data. We present these components, and give results and analyses for several open source, commercially-used algorithms. The goal for NAB is to provide a standard, open source framework with which the research community can compare and evaluate different algorithms for detecting anomalies in streaming data.

14th International Conference on Machine Learning and Applications (IEEE ICMLA), 2015. Fixed typo in equation and formatting

References in corpus (1)

• Finding Anomalous Periodic Time Series: An Application to Catalogs of Periodic Variable Stars

Cited by in corpus (66)

• Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding
• A Unifying Review of Deep and Shallow Anomaly Detection
• Continuous online sequence learning with an unsupervised neural network model
• TAnoGAN: Time Series Anomaly Detection with Generative Adversarial Networks
• RobustTAD: Robust Time Series Anomaly Detection via Decomposition and Convolutional Neural Networks
• Local Evaluation of Time Series Anomaly Detection Algorithms
• Real-Time Anomaly Detection for Streaming Analytics
• Anomaly Detection in Predictive Maintenance: A New Evaluation Framework for Temporal Unsupervised Anomaly Detection Algorithms
• Anomaly Detection for Water Treatment System based on Neural Network with Automatic Architecture Optimization
• RNN-based Early Cyber-Attack Detection for the Tennessee Eastman Process
• A Review of 40 Years of Cognitive Architecture Research: Core Cognitive Abilities and Practical Applications
• Is it worth it? Comparing six deep and classical methods for unsupervised anomaly detection in time series
• Expert enhanced dynamic time warping based anomaly detection
• ReRe: A Lightweight Real-time Ready-to-Go Anomaly Detection Approach for Time Series
• A Learning-Based Fast Uplink Grant for Massive IoT via Support Vector Machines and Long Short-Term Memory
• ASAP: Prioritizing Attention via Time Series Smoothing
• CARE to Compare: A real-world dataset for anomaly detection in wind turbine data
• No Free Lunch But A Cheaper Supper: A General Framework for Streaming Anomaly Detection
• Hybridization of Capsule and LSTM Networks for unsupervised anomaly detection on multivariate data
• Learning to Detect Anomalous Wireless Links in IoT Networks
• The model of an anomaly detector for HiLumi LHC magnets based on Recurrent Neural Networks and adaptive quantization
• Sintel: A Machine Learning Framework to Extract Insights from Signals
• A Delay Metric for Video Object Detection: What Average Precision Fails to Tell
• End-to-End Memristive HTM System for Pattern Recognition and Sequence Prediction
• Neuromemrisitive Architecture of HTM with On-Device Learning and Neurogenesis
• On the Runtime-Efficacy Trade-off of Anomaly Detection Techniques for Real-Time Streaming Data
• I/O in Machine Learning Applications on HPC Systems: A 360-degree Survey
• Exathlon: A Benchmark for Explainable Anomaly Detection over Time Series
• Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and Dataset
• Hardware Architecture Proposal for TEDA algorithm to Data Streaming Anomaly Detection
• My(o) Armband Leaks Passwords: An EMG and IMU Based Keylogging Side-Channel Attack
• Distribution Agnostic Symbolic Representations for Time Series Dimensionality Reduction and Online Anomaly Detection
• How Far Should We Look Back to Achieve Effective Real-Time Time-Series Anomaly Detection?
• Conformalized density- and distance-based anomaly detection in time-series data
• Precision and Recall for Range-Based Anomaly Detection
• ALGAN: Time Series Anomaly Detection with Adjusted-LSTM GAN
• OneFlow: One-class flow for anomaly detection based on a minimal volume region
• Towards a Near Universal Time Series Data Mining Tool: Introducing the Matrix Profile
• RSM-GAN: A Convolutional Recurrent GAN for Anomaly Detection in Contaminated Seasonal Multivariate Time Series
• NVAE-GAN Based Approach for Unsupervised Time Series Anomaly Detection
• Improving Robustness on Seasonality-Heavy Multivariate Time Series Anomaly Detection
• SoK: Evaluations in Industrial Intrusion Detection Research
• Latent Neural Stochastic Differential Equations for Change Point Detection
• Calculating the matrix profile from noisy data
• anomaly : Detection of Anomalous Structure in Time Series Data
• Should I Raise The Red Flag? A comprehensive survey of anomaly scoring methods toward mitigating false alarms
• PyODDS: An End-to-end Outlier Detection System with Automated Machine Learning
• CLAASIC: a Cortex-Inspired Hardware Accelerator
• Integrated Clustering and Anomaly Detection (INCAD) for Streaming Data (Revised)
• Feynman Machine: The Universal Dynamical Systems Computer
• Innovative And Additive Outlier Robust Kalman Filtering With A Robust Particle Filter
• Paranom: A Parallel Anomaly Dataset Generator
• A Hybrid Framework for Real-Time Data Drift and Anomaly Identification Using Hierarchical Temporal Memory and Statistical Tests
• Real-Time Anomaly Detection for Advanced Manufacturing: Improving on Twitter's State of the Art
• MSAD: A Deep Dive into Model Selection for Time series Anomaly Detection
• OrionBench: Benchmarking Time Series Generative Models in the Service of the End-User
• Novelty Detection in Sequential Data by Informed Clustering and Modeling
• Predictive maintenance on event logs: Application on an ATM fleet
• Kontrol Edilebilir ptSTL Formulu Sentezi -- Synthesis of Controllable ptSTL Formulas
• Sequential online prediction in the presence of outliers and change points: an instant temporal structure learning approach
• Pastprop-RNN: improved predictions of the future by correcting the past
• Comprehensible Counterfactual Explanation on Kolmogorov-Smirnov Test
• Adaptive conversion of real-valued input into spike trains
• Data-driven Design of Context-aware Monitors for Hazard Prediction in Artificial Pancreas Systems
• An Evaluation of Methods for Real-Time Anomaly Detection using Force Measurements from the Turning Process
• One-to-One Matching of RTT and Path Changes