Unsupervised detection of semantic correlations in big data

Abstract

In real-world data, information is stored in extremely large feature vectors. These variables are typically correlated due to complex interactions involving many features simultaneously. Such correlations qualitatively correspond to semantic roles and are naturally recognized by both the human brain and artificial neural networks. This recognition enables, for instance, the prediction of missing parts of an image or text based on their context. We present a method to detect these correlations in high-dimensional data represented as binary numbers. We estimate the binary intrinsic dimension of a dataset, which quantifies the minimum number of independent coordinates needed to describe the data, and is therefore a proxy of semantic complexity. The proposed algorithm is largely insensitive to the so-called curse of dimensionality, and can therefore be used in big data analysis. We test this approach identifying phase transitions in model magnetic systems and we then apply it to the detection of semantic correlations of images and text inside deep neural networks.

References in corpus (14)

• Statistical physics of social dynamics
• Quantum Spin Liquids
• Scaling Laws for Neural Language Models
• A Survey on Methods and Theories of Quantized Neural Networks
• Entropy and Long range correlations in literary English
• Universal and accessible entropy estimation using a compression algorithm
• Ranking the information content of distance measures
• DADApy: Distance-based Analysis of DAta-manifolds in Python
• Intrinsic dimension estimation for locally undersampled data
• BitNet: Scaling 1-bit Transformers for Large Language Models
• Intrinsic dimension estimation for discrete metrics
• What is the $\text{intrinsic}$ dimension of your binary data? -- and how to compute it quickly
• Phase transition in large language models and the criticality of natural languages
• Emergence of a High-Dimensional Abstraction Phase in Language Transformers

Cited by in corpus (1)

• Family-Vicsek universality of the binary intrinsic dimension of nonequilibrium data