Real-Time Nonparametric Anomaly Detection in High-Dimensional Settings

Abstract

Timely detection of abrupt anomalies is crucial for real-time monitoring and security of modern systems producing high-dimensional data. With this goal, we propose effective and scalable algorithms. Proposed algorithms are nonparametric as both the nominal and anomalous multivariate data distributions are assumed unknown. We extract useful univariate summary statistics and perform anomaly detection in a single-dimensional space. We model anomalies as persistent outliers and propose to detect them via a cumulative sum-like algorithm. In case the observed data have a low intrinsic dimensionality, we learn a submanifold in which the nominal data are embedded and evaluate whether the sequentially acquired data persistently deviate from the nominal submanifold. Further, in the general case, we learn an acceptance region for nominal data via Geometric Entropy Minimization and evaluate whether the sequentially observed data persistently fall outside the acceptance region. We provide an asymptotic lower bound and an asymptotic approximation for the average false alarm period of the proposed algorithm. Moreover, we provide a sufficient condition to asymptotically guarantee that the decision statistic of the proposed algorithm does not diverge in the absence of anomalies. Experiments illustrate the effectiveness of the proposed schemes in quick and accurate anomaly detection in high-dimensional settings.

References in corpus (6)

• N-BaIoT: Network-based Detection of IoT Botnet Attacks Using Deep Autoencoders
• Real-Time Detection of Hybrid and Stealthy Cyber-Attacks in Smart Grid
• Changepoint detection for high-dimensional time series with missing data
• Anomaly Detection with Score functions based on Nearest Neighbor Graphs
• Multi-Sensor Sequential Change Detection with Unknown Change Propagation Pattern
• Robust Sequential Change-Point Detection by Convex Optimization

Cited by in corpus (13)

• IoT Anomaly Detection Methods and Applications: A Survey
• Super forecasting the technological singularity risks from artificial intelligence
• New and emerging forms of data and technologies: literature and bibliometric review
• Multi-Sensor Sequential Change Detection with Unknown Change Propagation Pattern
• Review of the state of the art in autonomous artificial intelligence
• Sequential Change-point Detection for High-dimensional and non-Euclidean Data
• A review on minimax rates in change point detection and localisation
• A Log-Linear Non-Parametric Online Changepoint Detection Algorithm based on Functional Pruning
• A distribution-free change-point monitoring scheme in high-dimensional settings with application to industrial image surveillance
• Anomaly Detection for a Large Number of Streams: A Permutation-Based Higher Criticism Approach
• Simple Neighborhood Representative Pre-processing Boosts Outlier Detectors
• Sequential Multivariate Change Detection with Calibrated and Memoryless False Detection Rates
• Sparse Anomaly Detection Across Referentials: A Rank-Based Higher Criticism Approach