Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks

Abstract

We analyze the learning dynamics of infinitely wide neural networks with a finite sized bottle-neck. Unlike the neural tangent kernel limit, a bottleneck in an otherwise infinite width network al-lows data dependent feature learning in its bottle-neck representation. We empirically show that a single bottleneck in infinite networks dramatically accelerates training when compared to purely in-finite networks, with an improved overall performance. We discuss the acceleration phenomena by drawing similarities to infinitely wide deep linear models, where the acceleration effect of a bottleneck can be understood theoretically.

References in corpus (9)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• Tensor Programs III: Neural Matrix Laws
• On Random Kernels of Residual Architectures