The Emergence of Spectral Universality in Deep Networks

Abstract

Recent work has shown that tight concentration of the entire spectrum of singular values of a deep network's input-output Jacobian around one at initialization can speed up learning by orders of magnitude. Therefore, to guide important design choices, it is important to build a full theoretical understanding of the spectra of Jacobians at initialization. To this end, we leverage powerful tools from free probability theory to provide a detailed analytic understanding of how a deep network's Jacobian spectrum depends on various hyperparameters including the nonlinearity, the weight and bias distributions, and the depth. For a variety of nonlinearities, our work reveals the emergence of new universal limiting spectral distributions that remain concentrated around one even as the depth goes to infinity.

arXiv comment17 pages, 4 figures. Appearing at the 21st International Conference on Artificial Intelligence and Statistics (AISTATS) 2018

Cited by in corpus (46)

• ReZero is All You Need: Fast Convergence at Large Depth
• Optimization for deep learning: theory and algorithms
• An analytic theory of generalization dynamics and transfer learning in deep linear networks
• Products of Many Large Random Matrices and Gradients in Deep Neural Networks
• Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• A Fine-Grained Spectral Perspective on Neural Networks
• Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
• Understanding self-supervised Learning Dynamics without Contrastive Pairs
• Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
• A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks
• Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
• A Signal Propagation Perspective for Pruning Neural Networks at Initialization
• On Random Matrices Arising in Deep Neural Networks. Gaussian Case
• A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent
• Tensor Programs III: Neural Matrix Laws
• Beyond islands: A free probabilistic approach
• The Benefits of Over-parameterization at Initialization in Deep ReLU Networks
• On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization
• Effect of Activation Functions on the Training of Overparametrized Neural Nets
• The Low-Rank Simplicity Bias in Deep Networks
• Sublinear Time Spectral Density Estimation
• Pathological spectra of the Fisher information metric and its variants in deep neural networks
• The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks
• Interpreting Deep Learning: The Machine Learning Rorschach Test?
• Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
• Neural Anisotropy Directions
• Information Geometry of Orthogonal Initializations and Training
• Constraint-Based Regularization of Neural Networks
• Initialization of ReLUs for Dynamical Isometry
• Spectrum concentration in deep residual learning: a free probability approach
• Universal characteristics of deep neural network loss surfaces from random matrix theory
• Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Case
• Adversarial Robustness Guarantees for Random Deep Neural Networks
• Selective Forgetting of Deep Networks at a Finer Level than Samples
• Ridge Regression with Over-Parametrized Two-Layer Networks Converge to Ridgelet Spectrum
• Mean field theory for deep dropout networks: digging up gradient backpropagation deeply
• Infinitely deep neural networks as diffusion processes
• Beyond Folklore: A Scaling Calculus for the Design and Initialization of ReLU Networks
• Residual Networks as Nonlinear Systems: Stability Analysis using Linearization
• The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry
• Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks
• Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks
• Almost Sure Asymptotic Freeness of Neural Network Jacobian with Orthogonal Weights
• Periodic Spectral Ergodicity: A Complexity Measure for Deep Neural Networks and Neural Architecture Search
• Large-Dimensional Random Matrix Theory and Its Applications in Deep Learning and Wireless Communications