A Revision of Neural Tangent Kernel-based Approaches for Neural Networks

Abstract

Recent theoretical works based on the neural tangent kernel (NTK) have shed light on the optimization and generalization of over-parameterized networks, and partially bridge the gap between their practical success and classical learning theory. Especially, using the NTK-based approach, the following three representative results were obtained: (1) A training error bound was derived to show that networks can fit any finite training sample perfectly by reflecting a tighter characterization of training speed depending on the data complexity. (2) A generalization error bound invariant of network size was derived by using a data-dependent complexity measure (CMD). It follows from this CMD bound that networks can generalize arbitrary smooth functions. (3) A simple and analytic kernel function was derived as indeed equivalent to a fully-trained network. This kernel outperforms its corresponding network and the existing gold standard, Random Forests, in few shot learning. For all of these results to hold, the network scaling factor $\kappa$ should decrease w.r.t. sample size n. In this case of decreasing $\kappa$, however, we prove that the aforementioned results are surprisingly erroneous. It is because the output value of trained network decreases to zero when $\kappa$ decreases w.r.t. n. To solve this problem, we tighten key bounds by essentially removing $\kappa$-affected values. Our tighter analysis resolves the scaling problem and enables the validation of the original NTK-based results.

arXiv commentWe spotted an error in the proof of Lemma A.4 and are investigating whether this can be corrected. Furthermore, the authors of the original paper have informed us that they are fixing the lemma upon which our theorem 3.2 builds. Therefore, we are removing the current version of our paper

References in corpus (12)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
• Enhanced Convolutional Neural Tangent Kernels
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian
• Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections
• Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee
• A priori generalization error for two-layer ReLU neural network through minimum norm solution