Detecting deviating data cells

Abstract

A multivariate dataset consists of $n$ cases in $d$ dimensions, and is often stored in an $n$ by $d$ data matrix. It is well-known that real data may contain outliers. Depending on the situation, outliers may be (a) undesirable errors which can adversely affect the data analysis, or (b) valuable nuggets of unexpected information. In statistics and data analysis the word outlier usually refers to a row of the data matrix, and the methods to detect such outliers only work when at least half the rows are clean. But often many rows have a few contaminated cell values, which may not be visible by looking at each variable (column) separately. We propose the first method to detect deviating data cells in a multivariate sample which takes the correlations between the variables into account. It has no restriction on the number of clean rows, and can deal with high dimensions. Other advantages are that it provides estimates of the `expected' values of the outlying cells, while imputing missing values at the same time. We illustrate the method on several real data sets, where it uncovers more structure than found by purely columnwise methods or purely rowwise methods. The proposed method can help to diagnose why a certain row is outlying, e.g. in process control. It may also serve as an initial step for estimating multivariate location and scatter matrices.

To appear in Technometrics

References in corpus (3)

• Propagation of outliers in multivariate data
• The shooting S-estimator for robust regression
• Robust regression estimation and inference in the presence of cellwise and casewise contamination

Cited by in corpus (18)

• Fast robust correlation for high-dimensional data
• Transforming variables to central normality
• MacroPCA: An all-in-one PCA method allowing for missing values as well as cellwise and rowwise outliers
• ICS for Multivariate Outlier Detection with Application to Quality Control
• The Cellwise Minimum Covariance Determinant Estimator
• Robust Identification of Target Genes and Outliers in Triple-negative Breast Cancer Data
• Challenges of cellwise outliers
• Multivariate outlier explanations using Shapley values and Mahalanobis distances
• Robust Multivariate Functional Control Chart
• Outlyingness: why do outliers lie out?
• Robust Multivariate Estimation Based On Statistical Depth Filters
• Multivariate Singular Spectrum Analysis by Robust Diagonalwise Low-Rank Approximation
• Analyzing cellwise weighted data
• MacroPARAFAC for handling rowwise and cellwise outliers in incomplete multi-way data
• Correspondence analysis: handling cell-wise outliers via the reconstitution algorithm
• Robust Principal Components by Casewise and Cellwise Weighting
• Casewise and Cellwise Robust Multilinear Principal Component Analysis
• Data-Driven Random Projection and Screening for High-Dimensional Generalized Linear Models