Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width

Abstract

We propose \emph{Taylorized training} as an initiative towards better understanding neural network training at finite width. Taylorized training involves training the $k$-th order Taylor expansion of the neural network at initialization, and is a principled extension of linearized training---a recently proposed theory for understanding the success of deep learning. We experiment with Taylorized training on modern neural network architectures, and show that Taylorized training (1) agrees with full neural network training increasingly better as we increase $k$, and (2) can significantly close the performance gap between linearized and full training. Compared with linearized training, higher-order training works in more realistic settings such as standard parameterization and large (initial) learning rate. We complement our experiments with theoretical results showing that the approximation error of $k$-th order Taylorized models decay exponentially over $k$ in wide neural networks.

References in corpus (11)

• Wide Residual Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• On Lazy Training in Differentiable Programming
• Are All Layers Created Equal?
• Enhanced Convolutional Neural Tangent Kernels
• SGD Learns the Conjugate Kernel Class of the Network
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• Gradients as Features for Deep Representation Learning

Cited by in corpus (8)

• The Quantum Path Kernel: a Generalized Quantum Neural Tangent Kernel for Deep Quantum Machine Learning
• Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
• Towards Understanding Hierarchical Learning: Benefits of Neural Representations
• On Infinite-Width Hypernetworks
• Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity
• Neural Networks as Kernel Learners: The Silent Alignment Effect
• Understanding Deflation Process in Over-parametrized Tensor Decomposition
• Catapult Dynamics and Phase Transitions in Quadratic Nets