Information Theoretic Interpretation of Deep learning

Abstract

We interpret part of the experimental results of Shwartz-Ziv and Tishby [2017]. Inspired by these results, we established a conjecture of the dynamics of the machinary of deep neural network. This conjecture can be used to explain the counterpart result by Saxe et al. [2018].

arXiv comment17 pages, 7 figures

References in corpus (8)

• Opening the Black Box of Deep Neural Networks via Information
• Deep Variational Information Bottleneck
• Spectrally-normalized margin bounds for neural networks
• Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• A Hitting Time Analysis of Stochastic Gradient Langevin Dynamics
• Local Optimality and Generalization Guarantees for the Langevin Algorithm via Empirical Metastability
• Generalization of an Upper Bound on the Number of Nodes Needed to Achieve Linear Separability

Cited by in corpus (1)

• Seeing Convolution Through the Eyes of Finite Transformation Semigroup Theory: An Abstract Algebraic Interpretation of Convolutional Neural Networks