Entropy and mutual information in models of deep neural networks

Abstract

We examine a class of deep learning models with a tractable method to compute information-theoretic quantities. Our contributions are three-fold: (i) We show how entropies and mutual informations can be derived from heuristic statistical physics methods, under the assumption that weight matrices are independent and orthogonally-invariant. (ii) We extend particular cases in which this result is known to be rigorously exact by providing a proof for two-layers networks with Gaussian random weights, using the recently introduced adaptive interpolation method. (iii) We propose an experiment framework with generative models of synthetic datasets, on which we train deep neural networks with a weight constraint designed so that the assumption in (i) is verified during learning. We study the behavior of entropies and mutual informations throughout learning and conclude that, in the proposed setting, the relationship between compression and generalization remains elusive.

References in corpus (20)

• Estimating Mutual Information
• Message Passing Algorithms for Compressed Sensing
• The information bottleneck method
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Opening the Black Box of Deep Neural Networks via Information
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Statistical physics of inference: Thresholds and algorithms
• MINE: Mutual Information Neural Estimation
• InfoVAE: Information Maximizing Variational Autoencoders
• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
• Nonlinear Information Bottleneck
• Efficient supervised learning in networks with binary synapses
• Estimating Mixture Entropy with Pairwise Distances
• The Mutual Information in Random Linear Estimation
• Relevant sparse codes with variational information bottleneck
• The adaptive interpolation method for proving replica formulas. Applications to the Curie-Weiss and Wigner spike models
• Inference from correlated patterns: a unified theory for perceptron learning and linear vector channels
• Multi-Layer Generalized Linear Estimation
• Inference in Deep Networks in High Dimensions

Cited by in corpus (44)

• Machine learning and the physical sciences
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Channel Estimation for One-Bit Multiuser Massive MIMO Using Conditional GAN
• On Information Plane Analyses of Neural Network Classifiers -- A Review
• The Gaussian equivalence of generative models for learning with shallow neural networks
• Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm
• Machine learning in spectral domain
• Mean-field inference methods for neural networks
• Uncertainty Quantification and Deep Ensembles
• Multi-Layer Bilinear Generalized Approximate Message Passing
• Deep radiomic signature with immune cell markers predicts the survival of glioma patients
• Review-Based Domain Disentanglement without Duplicate Users or Contexts for Cross-Domain Recommendation
• Calibrating cosmological simulations with implicit likelihood inference using galaxy growth observables
• Conditional Mutual Information Neural Estimator
• Caveats for information bottleneck in deterministic scenarios
• Exact asymptotics for phase retrieval and compressed sensing with random generative priors
• Eigenvalue distribution of nonlinear models of random matrices
• On the training of sparse and dense deep neural networks: less parameters, same performance
• On the Separability of Classes with the Cross-Entropy Loss Function
• Deep learning via message passing algorithms based on belief propagation
• Sliced Mutual Information: A Scalable Measure of Statistical Dependence
• On the Maximum Mutual Information Capacity of Neural Architectures
• Strong replica symmetry for high-dimensional disordered log-concave Gibbs measures
• Renormalized Mutual Information for Artificial Scientific Discovery
• Estimating Differential Entropy under Gaussian Convolutions
• Interpolating between boolean and extremely high noisy patterns through Minimal Dense Associative Memories
• Concentration of multi-overlaps for random ferromagnetic spin models
• Entropy of Sharp Restart
• Inference in Multi-Layer Networks with Matrix-Valued Unknowns
• OOGAN: Disentangling GAN with One-Hot Sampling and Orthogonal Regularization
• ACMamba: Fast Unsupervised Anomaly Detection via An Asymmetrical Consensus State Space Model
• Neural-prior stochastic block model
• Random Forests for Adaptive Nearest Neighbor Estimation of Information-Theoretic Quantities
• Free energy of multi-layer generalized linear models
• Information Plane Analysis Visualization in Deep Learning via Transfer Entropy
• Generalization Error of Generalized Linear Models in High Dimensions
• Learning to Find Correlated Features by Maximizing Information Flow in Convolutional Neural Networks
• Information Bottleneck: Exact Analysis of (Quantized) Neural Networks
• What do CNN neurons learn: Visualization & Clustering
• Bounds on the rates of statistical divergences and mutual information via stochastic thermodynamics
• Explicit mutual information for simple networks and neurons with lognormal activities
• Understanding Neural Networks with Logarithm Determinant Entropy Estimator
• Inference in Spreading Processes with Neural-Network Priors
• Think Global, Act Local: Relating DNN generalisation and node-level SNR