A note on regularised NTK dynamics with an application to PAC-Bayesian training

Abstract

We establish explicit dynamics for neural networks whose training objective has a regularising term that constrains the parameters to remain close to their initial value. This keeps the network in a lazy training regime, where the dynamics can be linearised around the initialisation. The standard neural tangent kernel (NTK) governs the evolution during the training in the infinite-width limit, although the regularisation yields an additional term appears in the differential equation describing the dynamics. This setting provides an appropriate framework to study the evolution of wide networks trained to optimise generalisation objectives such as PAC-Bayes bounds, and hence potentially contribute to a deeper theoretical understanding of such networks.

References in corpus (22)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Deep Neural Networks as Gaussian Processes
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• A Note on the PAC Bayesian Theorem
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
• On the properties of variational approximations of Gibbs posteriors
• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
• Finite Versus Infinite Neural Networks: an Empirical Study
• Tighter risk certificates for neural networks
• Feature Learning in Infinite-Width Neural Networks
• Finite size corrections for neural network Gaussian processes
• Learning PAC-Bayes Priors for Probabilistic Neural Networks
• PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization
• PAC-Bayesian Contrastive Unsupervised Representation Learning
• Characterizing the Spectrum of the NTK via a Power Series Expansion
• Learning via Wasserstein-Based High Probability Generalisation Bounds
• Demystify Optimization and Generalization of Over-parameterized PAC-Bayesian Learning