Intrinsic dimension estimation for discrete metrics
arXiv:2207.09688 · doi:10.1103/PhysRevLett.130.067401
Abstract
Real world-datasets characterized by discrete features are ubiquitous: from categorical surveys to clinical questionnaires, from unweighted networks to DNA sequences. Nevertheless, the most common unsupervised dimensional reduction methods are designed for continuous spaces, and their use for discrete spaces can lead to errors and biases. In this letter we introduce an algorithm to infer the intrinsic dimension (ID) of datasets embedded in discrete spaces. We demonstrate its accuracy on benchmark datasets, and we apply it to analyze a metagenomic dataset for species fingerprinting, finding a surprisingly small ID, of order 2. This suggests that evolutive pressure acts on a low-dimensional manifold despite the high-dimensionality of sequences' space.
RevTeX4.2, 13 pages, 10 figures
References in corpus (2)
Cited by in corpus (7)
- DADApy: Distance-based Analysis of DAta-manifolds in Python
- Improved decision making with similarity based machine learning: Applications in chemistry
- Non-parametric learning critical behavior in Ising partition functions: PCA entropy and intrinsic dimension
- From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networks
- Unsupervised detection of semantic correlations in big data
- Density Estimation via Binless Multidimensional Integration
- A general framework for adaptive nonparametric dimensionality reduction