Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and Dataset
arXiv:2411.09047 · doi:10.1109/ICSE-SEIP66354.2025.00039
Abstract
As Large-Scale Cloud Systems (LCS) become increasingly complex, effective anomaly detection is critical for ensuring system reliability and performance. However, there is a shortage of large-scale, real-world datasets available for benchmarking anomaly detection methods. To address this gap, we introduce a new high-dimensional dataset from IBM Cloud, collected over 4.5 months from the IBM Cloud Console. This dataset comprises 39,365 rows and 117,448 columns of telemetry data. Additionally, we demonstrate the application of machine learning models for anomaly detection and discuss the key challenges faced in this process. This study and the accompanying dataset provide a resource for researchers and practitioners in cloud system monitoring. It facilitates more efficient testing of anomaly detection methods in real-world data, helping to advance the development of robust solutions to maintain the health and performance of large-scale cloud infrastructures.
Added the reproducibility package to version 2. To appear in proceedings of ICSE SEIP 2025
References in corpus (7)
- Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding
- A Unifying Review of Deep and Shallow Anomaly Detection
- Evaluating Real-time Anomaly Detection Algorithms - the Numenta Anomaly Benchmark
- Anomaly Detection in a Large-scale Cloud Platform
- Anomaly Detection in Cloud Components
- Architecture for Analysis of Streaming Data
- Online and Offline Analysis of Streaming Data