Automatic topography of high-dimensional data sets by non-parametric Density Peak clustering

Abstract

Data analysis in high-dimensional spaces aims at obtaining a synthetic description of a data set, revealing its main structure and its salient features. We here introduce an approach providing this description in the form of a topography of the data, namely a human-readable chart of the probability density from which the data are harvested. The approach is based on an unsupervised extension of Density Peak clustering and a non-parametric density estimator that measures the probability density in the manifold containing the data. This allows finding automatically the number and the height of the peaks of the probability density, and the depth of the "valleys" separating them. Importantly, the density estimator provides a measure of the error, which allows distinguishing genuine density peaks from density fluctuations due to finite sampling. The approach thus provides robust and visual information about the density peaks' height, their statistical reliability, and their hierarchical organization, offering a conceptually powerful extension of the standard clustering partitions. We show that this framework is particularly useful in the analysis of complex data sets.

arXiv commentThere is a Supplementary Information document in the ancillary files folder

References in corpus (2)

• Scikit-learn: Machine Learning in Python
• Estimating the intrinsic dimension of datasets by a minimal neighborhood information

Cited by in corpus (11)

• Guiding New Physics Searches with Unsupervised Learning
• DADApy: Distance-based Analysis of DAta-manifolds in Python
• Dynamical Landscape and Multistability of a Climate Model
• Hierarchical clustering that takes advantage of both density-peak and density-connectivity
• Beyond Local Structures In Critical Supercooled Water Through Unsupervised Learning
• A density peaks clustering algorithm with sparse search and K-d tree
• ZundEig: The Structure of the Proton in Liquid Water From Unsupervised Learning
• Automatic feature selection and weighting in molecular systems using Differentiable Information Imbalance
• Statistically Unbiased Free Energy Estimates from Biased Simulations
• Hierarchical nucleation in deep neural networks
• Density Estimation via Binless Multidimensional Integration