Information Theoretic Structure Learning with Confidence

Abstract

Information theoretic measures (e.g. the Kullback Liebler divergence and Shannon mutual information) have been used for exploring possibly nonlinear multivariate dependencies in high dimension. If these dependencies are assumed to follow a Markov factor graph model, this exploration process is called structure discovery. For discrete-valued samples, estimates of the information divergence over the parametric class of multinomial models lead to structure discovery methods whose mean squared error achieves parametric convergence rates as the sample size grows. However, a naive application of this method to continuous nonparametric multivariate models converges much more slowly. In this paper we introduce a new method for nonparametric structure discovery that uses weighted ensemble divergence estimators that achieve parametric convergence rates and obey an asymptotic central limit theorem that facilitates hypothesis testing and other types of statistical validation.

10 pages, 3 figures

References in corpus (8)

• Estimation of Rényi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs
• Ensemble estimation of multivariate f-divergence
• Learning loopy graphical models with latent variables: Efficient methods and guarantees
• Multivariate f-Divergence Estimation With Confidence
• Learning Latent Tree Graphical Models
• Demystifying Fixed k-Nearest Neighbor Information Estimators
• Information Theoretic Structure Learning with Confidence
• Exponential Concentration of a Density Functional Estimator

Cited by in corpus (5)

• Direct Estimation of Information Divergence Using Nearest Neighbor Ratios
• Information Theoretic Structure Learning with Confidence
• Ensemble Estimation of Generalized Mutual Information with Applications to Genomics
• Geometric Estimation of Multivariate Dependency
• Feature Selection for multi-labeled variables via Dependency Maximization