On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization

Abstract

The prevailing thinking is that orthogonal weights are crucial to enforcing dynamical isometry and speeding up training. The increase in learning speed that results from orthogonal initialization in linear networks has been well-proven. However, while the same is believed to also hold for nonlinear networks when the dynamical isometry condition is satisfied, the training dynamics behind this contention have not been thoroughly explored. In this work, we study the dynamics of ultra-wide networks across a range of architectures, including Fully Connected Networks (FCNs) and Convolutional Neural Networks (CNNs) with orthogonal initialization via neural tangent kernel (NTK). Through a series of propositions and lemmas, we prove that two NTKs, one corresponding to Gaussian weights and one to orthogonal weights, are equal when the network width is infinite. Further, during training, the NTK of an orthogonally-initialized infinite-width network should theoretically remain constant. This suggests that the orthogonal initialization cannot speed up training in the NTK (lazy training) regime, contrary to the prevailing thoughts. In order to explore under what circumstances can orthogonality accelerate training, we conduct a thorough empirical investigation outside the NTK regime. We find that when the hyper-parameters are set to achieve a linear regime in nonlinear activation, orthogonal initialization can improve the learning speed with a large learning rate or large depth.

arXiv commentrevised theorems and completed proofs

References in corpus (23)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Deep Neural Networks as Gaussian Processes
• On Exact Computation with an Infinitely Wide Neural Net
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Exponential expressivity in deep neural networks through transient chaos
• On Lazy Training in Differentiable Programming
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Multivariate normal approximation using exchangeable pairs
• The Emergence of Spectral Universality in Deep Networks
• Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
• The large learning rate phase of deep learning: the catapult mechanism
• Neural Tangents: Fast and Easy Infinite Neural Networks in Python
• A Mean Field Theory of Batch Normalization
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• On the linearity of large non-linear models: when and why the tangent kernel is constant
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function
• On the infinite width limit of neural networks with a standard parameterization
• Information Geometry of Orthogonal Initializations and Training
• Mean field theory for deep dropout networks: digging up gradient backpropagation deeply
• Constructing exchangeable pairs by diffusion on manifolds and its application

Cited by in corpus (8)

• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Finite Versus Infinite Neural Networks: an Empirical Study
• Towards NNGP-guided Neural Architecture Search
• Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspective
• Implicit bias of deep linear networks in the large learning rate phase
• Constructing exchangeable pairs by diffusion on manifolds and its application
• The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry
• On the validity of kernel approximations for orthogonally-initialized neural networks