On the validity of kernel approximations for orthogonally-initialized neural networks

Abstract

In this note we extend kernel function approximation results for neural networks with Gaussian-distributed weights to single-layer networks initialized using Haar-distributed random orthogonal matrices (with possible rescaling). This is accomplished using recent results from random matrix theory.

References in corpus (8)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
• On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization

Cited by in corpus (1)

• Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping