Think Global, Act Local: Relating DNN generalisation and node-level SNR

Abstract

The reasons behind good DNN generalisation remain an open question. In this paper we explore the problem by looking at the Signal-to-Noise Ratio of nodes in the network. Starting from information theory principles, it is possible to derive an expression for the SNR of a DNN node output. Using this expression we construct figures-of-merit that quantify how well the weights of a node optimise SNR (or, equivalently, information rate). Applying these figures-of-merit, we give examples indicating that weight sets that promote good SNR performance also exhibit good generalisation. In addition, we are able to identify the qualities of weight sets that exhibit good SNR behaviour and hence promote good generalisation. This leads to a discussion of how these results relate to network training and regularisation. Finally, we identify some ways that these observations can be used in training design.

15 pages, 5 figures; for associated colab files see http://github.com/pnorridge/think-global-act-local/settings

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Improving neural networks by preventing co-adaptation of feature detectors
• Learning deep representations by mutual information estimation and maximization
• Understanding deep learning requires rethinking generalization
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Learning Discrete Representations via Information Maximizing Self-Augmented Training
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Information-theoretic analysis of generalization capability of learning algorithms
• Entropy and mutual information in models of deep neural networks
• Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• Stochastic Mirror Descent on Overparameterized Nonlinear Models: Convergence, Implicit Regularization, and Generalization
• An Information-Theoretic View for Deep Learning
• Theory IIIb: Generalization in Deep Networks
• Chaining Mutual Information and Tightening Generalization Bounds
• Information Bottleneck and its Applications in Deep Learning