Time-Series Anomaly Detection Service at Microsoft

Abstract

Large companies need to monitor various metrics (for example, Page Views and Revenue) of their applications and services in real time. At Microsoft, we develop a time-series anomaly detection service which helps customers to monitor the time-series continuously and alert for potential incidents on time. In this paper, we introduce the pipeline and algorithm of our anomaly detection service, which is designed to be accurate, efficient and general. The pipeline consists of three major modules, including data ingestion, experimentation platform and online compute. To tackle the problem of time-series anomaly detection, we propose a novel algorithm based on Spectral Residual (SR) and Convolutional Neural Network (CNN). Our work is the first attempt to borrow the SR model from visual saliency detection domain to time-series anomaly detection. Moreover, we innovatively combine SR and CNN together to improve the performance of SR model. Our approach achieves superior experimental results compared with state-of-the-art baselines on both public datasets and Microsoft production data.

arXiv commentKDD 2019

References in corpus (2)

• Tutorial on Variational Autoencoders
• Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications

Cited by in corpus (44)

• Deep Learning for Time Series Anomaly Detection: A Survey
• DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detection
• MST-GAT: A Multimodal Spatial-Temporal Graph Attention Network for Time Series Anomaly Detection
• An Evaluation of Anomaly Detection and Diagnosis in Multivariate Time Series
• TFAD: A Decomposition Time Series Anomaly Detection Architecture with Time-Frequency Analysis
• Correlation-aware Spatial-Temporal Graph Learning for Multivariate Time-series Anomaly Detection
• RobustTAD: Robust Time Series Anomaly Detection via Decomposition and Convolutional Neural Networks
• Unsupervised Deep Learning for IoT Time Series
• A Joint Model for IT Operation Series Prediction and Anomaly Detection
• Adaptive Performance Anomaly Detection for Online Service Systems via Pattern Sketching
• Explainable Time Series Anomaly Detection using Masked Latent Generative Modeling
• ReRe: A Lightweight Real-time Ready-to-Go Anomaly Detection Approach for Time Series
• Multivariate Time-series Anomaly Detection via Graph Attention Network
• RLAD: Time Series Anomaly Detection through Reinforcement Learning and Active Learning
• Sintel: A Machine Learning Framework to Extract Insights from Signals
• Merlion: A Machine Learning Library for Time Series
• Detecting Anomalies within Smart Buildings using Do-It-Yourself Internet of Things
• TODS: An Automated Time Series Outlier Detection System
• PySAD: A Streaming Anomaly Detection Framework in Python
• Hardware Architecture Proposal for TEDA algorithm to Data Streaming Anomaly Detection
• How Far Should We Look Back to Achieve Effective Real-Time Time-Series Anomaly Detection?
• AlerTiger: Deep Learning for AI Model Health Monitoring at LinkedIn
• Is Your Anomaly Detector Ready for Change? Adapting AIOps Solutions to the Real World
• NVAE-GAN Based Approach for Unsupervised Time Series Anomaly Detection
• Series Saliency: Temporal Interpretation for Multivariate Time Series Forecasting
• Anomaly Detection on Seasonal Metrics via Robust Time Series Decomposition
• Little Help Makes a Big Difference: Leveraging Active Learning to Improve Unsupervised Time Series Anomaly Detection
• Anomaly Detection on IT Operation Series via Online Matrix Profile
• Statistical Evaluation of Anomaly Detectors for Sequences
• Automated Model Selection for Time-Series Anomaly Detection
• Automated Contrastive Learning Strategy Search for Time Series
• Moving Metric Detection and Alerting System at eBay
• Lumos: A Library for Diagnosing Metric Regressions in Web-Scale Applications
• Unsupervised detection and open-set classification of fast-ramped flexibility activation events
• "I see models being a whole other thing": An Empirical Study of Pre-Trained Model Naming Conventions and A Tool for Enhancing Naming Consistency
• T-Rep: Representation Learning for Time Series using Time-Embeddings
• Diagnosing and Resolving Cloud Platform Instability with Multi-modal RAG LLMs
• PRAXIS: Integrating Program Analysis with Observability for Root-Cause Analysis
• Fast Partial Fourier Transform
• Detecting Anomalous LAN Activities under Differential Privacy
• OrionBench: Benchmarking Time Series Generative Models in the Service of the End-User
• A Semi-Supervised Approach for Abnormal Event Prediction on Large Operational Network Time-Series Data
• Comprehensible Counterfactual Explanation on Kolmogorov-Smirnov Test
• Revisiting Backdoor Attacks on Time Series Classification in the Frequency Domain