On perfect clustering of high dimension, low sample size data

Abstract

Popular clustering algorithms based on usual distance functions (e.g., Euclidean distance) often suffer in high dimension, low sample size (HDLSS) situations, where concentration of pairwise distances has adverse effects on their performance. In this article, we use a dissimilarity measure based on the data cloud, called MADD, which takes care of this problem. MADD uses the distance concentration phenomenon to its advantage, and as a result, clustering algorithms based on MADD usually perform better for high dimensional data. Using theoretical and numerical results, we amply demonstrate it in this article. We also address the problem of estimating the number of clusters. This is a very challenging problem in cluster analysis, and several algorithms have been proposed for it. We show that many of these existing algorithms have superior performance in high dimensions when MADD is used instead of the Euclidean distance. We also construct a new estimator based on penalized Dunn index and prove its consistency in the HDLSS asymptotic regime, where the sample size remains fixed and the dimension grows to infinity. Several simulated and real data sets are analyzed to demonstrate the importance of MADD for cluster analysis of high dimensional data.

References in corpus (1)

• Natural Scales in Geographical Patterns

Cited by in corpus (5)

• FedGroup: Efficient Clustered Federated Learning via Decomposed Data-Driven Measure
• Flexible Clustered Federated Learning for Client-Level Data Distribution Shift
• Classification Using Global and Local Mahalanobis Distances
• Methods for Quantifying Dataset Similarity: a Review, Taxonomy and Comparison
• On high-dimensional modifications of some graph-based two-sample tests