Scale-invariant representation of machine learning

Abstract

The success of machine learning has resulted from its structured representation of data. Similar data have close internal representations as compressed codes for classification or emerged labels for clustering. We observe that the frequency of internal codes or labels follows power laws in both supervised and unsupervised learning models. This scale-invariant distribution implies that machine learning largely compresses frequent typical data, and simultaneously, differentiates many atypical data as outliers. In this study, we derive the process by which these power laws can naturally arise in machine learning. In terms of information theory, the scale-invariant representation corresponds to a maximally uncertain data grouping among possible representations that guarantee a given learning accuracy.

References in corpus (14)

• Power laws, Pareto distributions and Zipf's law
• Machine learning and the physical sciences
• Machine learning phases of matter
• Learning phase transitions by confusion
• Identifying Quantum Phase Transitions using Artificial Neural Networks on Experimental Data
• Power-law statistics and universal scaling in the absence of criticality
• Information-based fitness and the emergence of criticality in living systems
• Zipf's law and criticality in multivariate data without fine-tuning
• On sampling and modeling complex systems
• Disentangling feature and lazy training in deep neural networks
• Statistical Criticality arises in Most Informative Representations
• Criticality of mostly informative samples: A Bayesian model selection approach
• Zipf's and Taylor's Laws
• Optimal encoding in stochastic latent-variable Models