A Fine-Grained Spectral Perspective on Neural Networks

Abstract

Are neural networks biased toward simple functions? Does depth always help learn more complex features? Is training the last layer of a network as good as training all layers? How to set the range for learning rate tuning? These questions seem unrelated at face value, but in this work we give all of them a common treatment from the spectral perspective. We will study the spectra of the *Conjugate Kernel, CK,* (also called the *Neural Network-Gaussian Process Kernel*), and the *Neural Tangent Kernel, NTK*. Roughly, the CK and the NTK tell us respectively "what a network looks like at initialization" and "what a network looks like during and after training." Their spectra then encode valuable information about the initial distribution and the training and generalization properties of neural networks. By analyzing the eigenvalues, we lend novel insights into the questions put forth at the beginning, and we verify these insights by extensive experiments of neural networks. We derive fast algorithms for computing the spectra of CK and NTK when the data is uniformly distributed over the boolean cube, and show this spectra is the same in high dimensions when data is drawn from isotropic Gaussian or uniformly over the sphere. Code replicating our results is available at github.com/thegregyang/NNspectra.

arXiv comment8 pages of main text, 19 figures, 51 pages including appendix

References in corpus (27)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Deep Neural Networks as Gaussian Processes
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Strictly and non-strictly positive definite functions on spheres
• Exponential expressivity in deep neural networks through transient chaos
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks
• The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
• On the Convergence Rate of Training Recurrent Neural Networks
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Stochastic Variational Deep Kernel Learning
• Deep Convolutional Networks as shallow Gaussian Processes
• The Emergence of Spectral Universality in Deep Networks
• Towards Characterizing Divergence in Deep Q-Learning
• How to Start Training: The Effect of Initialization and Architecture
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• Training behavior of deep neural network in frequency domain
• On the Inductive Bias of Neural Tangent Kernels
• On the Selection of Initialization and Activation Function for Deep Neural Networks
• Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?
• Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
• Deep Gaussian Processes with Convolutional Kernels
• A Gaussian Process perspective on Convolutional Neural Networks

Cited by in corpus (21)

• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• Neural Networks and Quantum Field Theory
• Towards Understanding the Spectral Bias of Deep Learning
• Finite Versus Infinite Neural Networks: an Empirical Study
• Star-Galaxy Image Separation with Computationally Efficient Gaussian Process Classification
• Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks
• The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks
• Expressivity of Variational Quantum Machine Learning on the Boolean Cube
• Generalization bounds for deep learning
• Neural networks are a priori biased towards Boolean functions with low entropy
• Pathological spectra of the Fisher information metric and its variants in deep neural networks
• Deep Equals Shallow for ReLU Networks in Kernel Regimes
• Space of Functions Computed by Deep-Layered Machines
• Is SGD a Bayesian sampler? Well, almost
• Deep frequency principle towards understanding why deeper learning is faster
• Why flatness does and does not correlate with generalization for deep neural networks
• An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network
• Learning curves for Gaussian process regression with power-law priors and targets
• Large-Dimensional Random Matrix Theory and Its Applications in Deep Learning and Wireless Communications
• Constrained Policy Gradient Method for Safe and Fast Reinforcement Learning: a Neural Tangent Kernel Based Approach
• Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks