A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks

Abstract

A recent breakthrough in deep learning theory shows that the training of over-parameterized deep neural networks can be characterized by a kernel function called \textit{neural tangent kernel} (NTK). However, it is known that this type of results does not perfectly match the practice, as NTK-based analysis requires the network weights to stay very close to their initialization throughout training, and cannot handle regularizers or gradient noises. In this paper, we provide a generalized neural tangent kernel analysis and show that noisy gradient descent with weight decay can still exhibit a "kernel-like" behavior. This implies that the training loss converges linearly up to a certain accuracy. We also establish a novel generalization error bound for two-layer neural networks trained by noisy gradient descent with weight decay.

arXiv comment33 pages. This version changes the title and improves the presentation

References in corpus (28)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• On Exact Computation with an Infinitely Wide Neural Net
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Convergence Analysis of Two-layer Neural Networks with ReLU Activation
• On Lazy Training in Differentiable Programming
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• Recovery Guarantees for One-hidden-layer Neural Networks
• Learning ReLUs via Gradient Descent
• Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
• Norm-Based Capacity Control in Neural Networks
• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
• An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis
• Towards Understanding the Spectral Bias of Deep Learning
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• Learning One-hidden-layer ReLU Networks via Gradient Descent
• Implicit Bias of Gradient Descent on Linear Convolutional Networks
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
• On Learning Over-parameterized Neural Networks: A Functional Approximation Perspective
• Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models
• Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems
• Tight Sample Complexity of Learning One-hidden-layer Convolutional Neural Networks
• Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks
• Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations

Cited by in corpus (15)

• Explicit Regularisation in Gaussian Noise Injections
• Mathematical Models of Overparameterized Neural Networks
• When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?
• Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks
• Efficient Sparse Coding using Hierarchical Riemannian Pursuit
• Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theory
• A Spectral Analysis of Dot-product Kernels
• When does gradient descent with logistic loss find interpolating two-layer networks?
• Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft Committee Machines
• On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime
• A Fully Spiking Hybrid Neural Network for Energy-Efficient Object Detection
• Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysis
• Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity
• Neural Tangent Kernel Empowered Federated Learning
• One-pass Stochastic Gradient Descent in Overparametrized Two-layer Neural Networks