Discovering Structure in High-Dimensional Data Through Correlation Explanation

Abstract

We introduce a method to learn a hierarchy of successively more abstract representations of complex data based on optimizing an information-theoretic objective. Intuitively, the optimization searches for a set of latent factors that best explain the correlations in the data as measured by multivariate mutual information. The method is unsupervised, requires no model assumptions, and scales linearly with the number of variables which makes it an attractive approach for very high dimensional systems. We demonstrate that Correlation Explanation (CorEx) automatically discovers meaningful structure for data from diverse sources including personality tests, DNA, and human language.

15 pages, 6 figures. Includes supplementary material and link to code. Published in the proceedings of the 28th Annual Conference on Neural Information Processing Systems, NIPS 2014

References in corpus (9)

• Scikit-learn: Machine Learning in Python
• The information bottleneck method
• Network information and connected correlations
• Nonnegative Decomposition of Multivariate Information
• A Survey on Latent Tree Models and Applications
• Learning the Structure of Deep Sparse Graphical Models
• Spectral Methods for Learning Multivariate Latent Tree Structure
• Maximally Informative Hierarchical Representations of High-Dimensional Data
• Information-theoretic inference of common ancestors

Cited by in corpus (10)

• Functional Connectome of the Human Brain with Total Correlation
• Demystifying Fixed k-Nearest Neighbor Information Estimators
• An Information-theoretic Approach to Unsupervised Feature Selection for High-Dimensional Data
• Info-Clustering: A Mathematical Theory for Data Clustering
• Toward Interpretable Topic Discovery via Anchored Correlation Explanation
• DEMI: Discriminative Estimator of Mutual Information
• The Design of Global Correlation Quantifiers and Continuous Notions of Statistical Sufficiency
• Correlation Congruence for Knowledge Distillation
• Analyzing Social Media Data to Understand Consumers' Information Needs on Dietary Supplements
• Building Models for Biopathway Dynamics Using Intrinsic Dimensionality Analysis