Sparse Anomaly Detection Across Referentials: A Rank-Based Higher Criticism Approach

Abstract

Detecting anomalies in large sets of observations is crucial in various applications, such as epidemiological studies, gene expression studies, and systems monitoring. We consider settings where the units of interest result in multiple independent observations from potentially distinct referentials. Scan statistics and related methods are commonly used in such settings, but rely on stringent modeling assumptions for proper calibration. We instead propose a rank-based variant of the higher criticism statistic that only requires independent observations originating from ordered spaces. We show under what conditions the resulting methodology is able to detect the presence of anomalies. These conditions are stated in a general, non-parametric manner, and depend solely on the probabilities of anomalous observations exceeding nominal observations. The analysis requires a refined understanding of the distribution of the ranks under the presence of anomalies, and in particular of the rank-induced dependencies. The methodology is robust against heavy-tailed distributions through the use of ranks. Within the exponential family and a family of convolutional models, we analytically quantify the asymptotic performance of our methodology and the performance of the oracle, and show the difference is small for many common models. Simulations confirm these results. We show the applicability of the methodology through an analysis of quality control data of a pharmaceutical manufacturing process.

References in corpus (17)

• Higher criticism for detecting sparse heterogeneous mixtures
• Global testing under sparse alternatives: ANOVA, multiple comparisons and the higher criticism
• Detection of an anomalous cluster in a network
• Innovated higher criticism for detecting sparse signals in correlated noise
• Higher Criticism for Large-Scale Inference, Especially for Rare and Weak Effects
• Computational barriers in minimax submatrix detection
• Detection of a sparse submatrix of a high-dimensional noisy matrix
• Optimal and fast detection of spatial clusters with scan statistics
• Properties of higher criticism under strong dependence
• Real-Time Nonparametric Anomaly Detection in High-Dimensional Settings
• Estimation and confidence sets for sparse normal mixtures
• On the exact Berk-Jones statistics and their p-value calculation
• Cosmological non-Gaussian Signature Detection: Comparing Performance of Different Statistical Tests
• Hypothesis testing for high-dimensional sparse binary regression
• Detection boundary and Higher Criticism approach for rare and weak genetic effects
• Anomaly Detection for a Large Number of Streams: A Permutation-Based Higher Criticism Approach
• Sparse Anomaly Detection Across Referentials: A Rank-Based Higher Criticism Approach

Cited by in corpus (1)

• Sparse Anomaly Detection Across Referentials: A Rank-Based Higher Criticism Approach