Understanding Neural Networks with Logarithm Determinant Entropy Estimator

Abstract

Understanding the informative behaviour of deep neural networks is challenged by misused estimators and the complexity of network structure, which leads to inconsistent observations and diversified interpretation. Here we propose the LogDet estimator -- a reliable matrix-based entropy estimator that approximates Shannon differential entropy. We construct informative measurements based on LogDet estimator, verify our method with comparable experiments and utilize it to analyse neural network behaviour. Our results demonstrate the LogDet estimator overcomes the drawbacks that emerge from highly diverse and degenerated distribution thus is reliable to estimate entropy in neural networks. The Network analysis results also find a functional distinction between shallow and deeper layers, which can help understand the compression phenomenon in the Information bottleneck theory of neural networks.

arXiv comment15pages,22 figures

References in corpus (14)

• Estimating Mutual Information
• The information bottleneck method
• Opening the Black Box of Deep Neural Networks via Information
• Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle
• Entropy and mutual information in models of deep neural networks
• Law of Log Determinant of Sample Covariance Matrix and Optimal Estimation of Differential Entropy for High-Dimensional Gaussian Distributions
• Log-Determinant Divergences Revisited: Alpha--Beta and Gamma Log-Det Divergences
• On Information Plane Analyses of Neural Network Classifiers -- A Review
• Strong subadditivity for log-determinant of covariance matrices and its applications
• Deep Networks from the Principle of Rate Reduction
• Measuring Dependence with Matrix-based Entropy Functional
• Deep Deterministic Information Bottleneck with Matrix-based Entropy Functional
• Do Compressed Representations Generalize Better?
• Incremental Learning via Rate Reduction