Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity

Abstract

As a popular approach to modeling the dynamics of training overparametrized neural networks (NNs), the neural tangent kernels (NTK) are known to fall behind real-world NNs in generalization ability. This performance gap is in part due to the \textit{label agnostic} nature of the NTK, which renders the resulting kernel not as \textit{locally elastic} as NNs~\citep{he2019local}. In this paper, we introduce a novel approach from the perspective of \emph{label-awareness} to reduce this gap for the NTK. Specifically, we propose two label-aware kernels that are each a superimposition of a label-agnostic part and a hierarchy of label-aware parts with increasing complexity of label dependence, using the Hoeffding decomposition. Through both theoretical and empirical evidence, we show that the models trained with the proposed kernels better simulate NNs in terms of generalization ability and local elasticity.

NeurIPS 2020 camera ready version, 32 pages, 2 figures, 3 tables

References in corpus (19)

• Towards A Rigorous Science of Interpretable Machine Learning
• The information bottleneck method
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Deep Neural Networks as Gaussian Processes
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• On Lazy Training in Differentiable Programming
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Deep Convolutional Networks as shallow Gaussian Processes
• Enhanced Convolutional Neural Tangent Kernels
• Stiffness: A New Perspective on Generalization in Neural Networks
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
• Finite Depth and Width Corrections to the Neural Tangent Kernel
• Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks
• Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width

Cited by in corpus (4)

• Finite-time Lyapunov exponents of deep neural networks
• Toward Better Generalization Bounds with Locally Elastic Stability
• Uniform Convergence, Adversarial Spheres and a Simple Remedy
• A Modular Analysis of Provable Acceleration via Polyak's Momentum: Training a Wide ReLU Network and a Deep Linear Network