Traditional and Heavy-Tailed Self Regularization in Neural Network Models

Abstract

Random Matrix Theory (RMT) is applied to analyze the weight matrices of Deep Neural Networks (DNNs), including both production quality, pre-trained models such as AlexNet and Inception, and smaller models trained from scratch, such as LeNet5 and a miniature-AlexNet. Empirical and theoretical results clearly indicate that the empirical spectral density (ESD) of DNN layer matrices displays signatures of traditionally-regularized statistical models, even in the absence of exogenously specifying traditional forms of regularization, such as Dropout or Weight Norm constraints. Building on recent results in RMT, most notably its extension to Universality classes of Heavy-Tailed matrices, we develop a theory to identify \emph{5+1 Phases of Training}, corresponding to increasing amounts of \emph{Implicit Self-Regularization}. For smaller and/or older DNNs, this Implicit Self-Regularization is like traditional Tikhonov regularization, in that there is a `size scale' separating signal from noise. For state-of-the-art DNNs, however, we identify a novel form of \emph{Heavy-Tailed Self-Regularization}, similar to the self-organization seen in the statistical physics of disordered systems. This implicit Self-Regularization can depend strongly on the many knobs of the training process. By exploiting the generalization gap phenomena, we demonstrate that we can cause a small model to exhibit all 5+1 phases of training simply by changing the batch size.

Very abridged version of arXiv:1810.01075

References in corpus (12)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Noise Dressing of Financial Correlation Matrices
• Problems with Fitting to the Power-Law Distribution
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Universality Classes for Extreme Value Statistics
• Power-law distributions in binned empirical data
• Regularization for Deep Learning: A Taxonomy
• Rational Decisions, Random Matrices and Spin Glasses
• On the top eigenvalue of heavy-tailed random matrices
• Fitting Power-laws in empirical data with estimators that work for all exponents
• Beyond universality in random matrix theory
• Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior

Cited by in corpus (14)

• DeepCOVIDExplainer: Explainable COVID-19 Diagnosis Based on Chest X-ray Images
• Low-Memory Neural Network Training: A Technical Report
• Multiplicative noise and heavy tails in stochastic optimization
• On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks
• Resilient Cyberphysical Systems and their Application Drivers: A Technology Roadmap
• Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks
• Stable behaviour of infinitely wide deep neural networks
• GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training
• Train-by-Reconnect: Decoupling Locations of Weights from their Values
• Fractional moment-preserving initialization schemes for training deep neural networks
• Infinitely deep neural networks as diffusion processes
• An Investigation of the Weight Space to Monitor the Training Progress of Neural Networks
• Periodic Spectral Ergodicity: A Complexity Measure for Deep Neural Networks and Neural Architecture Search
• Eliminating Sharp Minima from SGD with Truncated Heavy-tailed Noise