Estimating the intrinsic dimension of datasets by a minimal neighborhood information

Abstract

Analyzing large volumes of high-dimensional data is an issue of fundamental importance in data science, molecular simulations and beyond. Several approaches work on the assumption that the important content of a dataset belongs to a manifold whose Intrinsic Dimension (ID) is much lower than the crude large number of coordinates. Such manifold is generally twisted and curved, in addition points on it will be non-uniformly distributed: two factors that make the identification of the ID and its exploitation really hard. Here we propose a new ID estimator using only the distance of the first and the second nearest neighbor of each point in the sample. This extreme minimality enables us to reduce the effects of curvature, of density variation, and the resulting computational cost. The ID estimator is theoretically exact in uniformly distributed datasets, and provides consistent measures in general. When used in combination with block analysis, it allows discriminating the relevant dimensions as a function of the block size. This allows estimating the ID even when the data lie on a manifold perturbed by a high-dimensional noise, a situation often encountered in real world data sets. We demonstrate the usefulness of the approach on molecular simulations and image analysis.

Scientific Reports 2017

References in corpus (1)

• Predicting the kinetics of RNA oligonucleotides using Markov state models

Cited by in corpus (72)

• Shallow Neural Networks for Fluid Flow Reconstruction with Limited Sensors
• Intrinsic dimension of data representations in deep neural networks
• Dissipative Floquet Dynamics: from Steady State to Measurement Induced Criticality in Trapped-ion Chains
• Scikit-dimension: a Python package for intrinsic dimension estimation
• Automatic topography of high-dimensional data sets by non-parametric Density Peak clustering
• Measurement-induced criticality as a data-structure transition
• Patch SVDD: Patch-level SVDD for Anomaly Detection and Segmentation
• Unfolding Hidden Barriers by Active Enhanced Sampling
• Roughness of molecular property landscapes and its impact on modellability
• Unsupervised learning universal critical behavior via the intrinsic dimension
• DADApy: Distance-based Analysis of DAta-manifolds in Python
• Dimensionality compression and expansion in Deep Neural Networks
• Dynamical Landscape and Multistability of a Climate Model
• Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm
• Exploring the robust extrapolation of high-dimensional machine learning potentials
• The mechanism of RNA base fraying: molecular dynamics simulations analyzed with core-set Markov state models
• Intrinsic dimension of path integrals: data mining quantum criticality and emergent simplicity
• Intrinsic dimension estimation for locally undersampled data
• Beyond Local Structures In Critical Supercooled Water Through Unsupervised Learning
• A Contrastive Variational Graph Auto-Encoder for Node Clustering
• TimeSOAP: Tracking high-dimensional fluctuations in complex molecular systems via time-variations of SOAP spectra
• Statistical learning theory of structured data
• The Intrinsic Dimension of Images and Its Impact on Learning
• Probing transfer learning with a model of synthetic correlated datasets
• Geometric compression of invariant manifolds in neural nets
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Domain Adaptation Principal Component Analysis: base linear method for learning with out-of-distribution data
• Counting the learnable functions of structured data
• A Neural Scaling Law from the Dimension of the Data Manifold
• ZundEig: The Structure of the Proton in Liquid Water From Unsupervised Learning
• 1-DREAM: 1D Recovery, Extraction and Analysis of Manifolds in noisy environments
• From observations to complexity of quantum states via unsupervised learning
• Rdimtools: An R package for Dimension Reduction and Intrinsic Dimension Estimation
• Automatic feature selection and weighting in molecular systems using Differentiable Information Imbalance
• Improved decision making with similarity based machine learning: Applications in chemistry
• Wave function network description and Kolmogorov complexity of quantum many-body systems
• Non-parametric learning critical behavior in Ising partition functions: PCA entropy and intrinsic dimension
• Genetic Programming for Manifold Learning: Preserving Local Topology
• Data-driven discovery of statistically relevant information in quantum simulators
• Investigating Berezinskii-Kosterlitz-Thouless phase transitions in Kagome spin ice by quantifying Monte Carlo process: Distribution of Hamming distances
• Steering-induced phase transition in measurement-only quantum circuits
• IAN: Iterated Adaptive Neighborhoods for manifold learning and dimensionality estimation
• Statistically Unbiased Free Energy Estimates from Biased Simulations
• The Intrinsic Manifolds of Radiological Images and their Role in Deep Learning
• Local intrinsic dimensionality estimators based on concentration of measure
• Sampling a rare protein transition with a hybrid classical-quantum computing algorithm
• Tensor network to learn the wavefunction of data
• Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisation
• Compressing and forecasting atomic material simulations with descriptors
• Relevant, hidden, and frustrated information in high-dimensional analyses of complex dynamical systems with internal noise
• Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks
• Distributional Results for Model-Based Intrinsic Dimension Estimators
• Intrinsic Geometric Vulnerability of High-Dimensional Artificial Intelligence
• Spectral Roll-off Points Variations: Exploring Useful Information in Feature Maps by Its Variations
• Graph Regularized Autoencoder and its Application in Unsupervised Anomaly Detection
• Dynamic Exploration Graph: A Novel Approach for Efficient Nearest Neighbor Search in Evolving Multimedia Datasets
• Reversible Gromov-Monge Sampler for Simulation-Based Inference
• Optimal 1-Wasserstein Distance for WGANs
• Complexity of Quantum Trajectories
• Data analysis from empirical moments and the Christoffel function
• The cell as a token: high-dimensional geometry in language models and cell embeddings
• On the criticality of the configuration-space statistical geometry
• Manifold-adaptive dimension estimation revisited
• Density Estimation via Binless Multidimensional Integration
• On Data-centric Myths
• Linking average- and worst-case perturbation robustness via class selectivity and dimensionality
• Deep Autoencoders: From Understanding to Generalization Guarantees
• Bias-corrected estimator for intrinsic dimension and differential entropy--a visual multiscale approach
• How isotropic kernels perform on simple invariants
• Cyclic image generation using chaotic dynamics
• Analysis of GSI HADES beamline configuration space
• MDIntrinsicDimension: Dimensionality-Based Analysis of Collective Motions in Macromolecules from Molecular Dynamics Trajectories