On the Inductive Bias of Neural Tangent Kernels

Abstract

State-of-the-art neural networks are heavily over-parameterized, making the optimization algorithm a crucial ingredient for learning predictive models with good generalization properties. A recent line of work has shown that in a certain over-parameterized regime, the learning dynamics of gradient descent are governed by a certain kernel obtained at initialization, called the neural tangent kernel. We study the inductive bias of learning in such a regime by analyzing this kernel and the corresponding function space (RKHS). In particular, we study smoothness, approximation, and stability properties of functions with finite norm, including stability to image deformations in the case of convolutional networks, and compare to other known kernels for similar architectures.

arXiv commentNeurIPS 2019

Cited by in corpus (12)

• Combining Machine Learning and Computational Chemistry for Predictive Insights Into Chemical Systems
• A Theoretical Analysis of Deep Q-Learning
• Convergence of Adversarial Training in Overparametrized Neural Networks
• A Fine-Grained Spectral Perspective on Neural Networks
• Finite Depth and Width Corrections to the Neural Tangent Kernel
• SINR: Deconvolving Circular SAS Images Using Implicit Neural Representations
• Locality defeats the curse of dimensionality in convolutional teacher-student scenarios
• On the infinite width limit of neural networks with a standard parameterization
• Convex Geometry and Duality of Over-parameterized Neural Networks
• Relative stability toward diffeomorphisms indicates performance in deep nets
• Out-of-Distribution Generalization in Kernel Regression
• Random matrices in service of ML footprint: ternary random features with no performance loss