On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization

Abstract

The prevailing thinking is that orthogonal weights are crucial to enforcing dynamical isometry and speeding up training. The increase in learning speed that results from orthogonal initialization in linear networks has been well-proven. However, while the same is believed to also hold for nonlinear networks when the dynamical isometry condition is satisfied, the training dynamics behind this contention have not been thoroughly explored. In this work, we study the dynamics of ultra-wide networks across a range of architectures, including Fully Connected Networks (FCNs) and Convolutional Neural Networks (CNNs) with orthogonal initialization via neural tangent kernel (NTK). Through a series of propositions and lemmas, we prove that two NTKs, one corresponding to Gaussian weights and one to orthogonal weights, are equal when the network width is infinite. Further, during training, the NTK of an orthogonally-initialized infinite-width network should theoretically remain constant. This suggests that the orthogonal initialization cannot speed up training in the NTK (lazy training) regime, contrary to the prevailing thoughts. In order to explore under what circumstances can orthogonality accelerate training, we conduct a thorough empirical investigation outside the NTK regime. We find that when the hyper-parameters are set to achieve a linear regime in nonlinear activation, orthogonal initialization can improve the learning speed with a large learning rate or large depth.

revised theorems and completed proofs

References in corpus (9)

• The large learning rate phase of deep learning: the catapult mechanism
• Neural Tangents: Fast and Easy Infinite Neural Networks in Python
• A Mean Field Theory of Batch Normalization
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• On the linearity of large non-linear models: when and why the tangent kernel is constant
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• On the infinite width limit of neural networks with a standard parameterization
• Mean field theory for deep dropout networks: digging up gradient backpropagation deeply
• Constructing exchangeable pairs by diffusion on manifolds and its application

Cited by in corpus (8)

• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Finite Versus Infinite Neural Networks: an Empirical Study
• Towards NNGP-guided Neural Architecture Search
• Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspective
• Implicit bias of deep linear networks in the large learning rate phase
• Constructing exchangeable pairs by diffusion on manifolds and its application
• The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry
• On the validity of kernel approximations for orthogonally-initialized neural networks