Every Model Learned by Gradient Descent Is Approximately a Kernel Machine

Abstract

Deep learning's successes are often attributed to its ability to automatically discover new representations of the data, rather than relying on handcrafted features like other learning methods. We show, however, that deep networks learned by the standard gradient descent algorithm are in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel). This greatly enhances the interpretability of deep network weights, by elucidating that they are effectively a superposition of the training examples. The network architecture incorporates knowledge of the target function into the kernel. This improved understanding should lead to better learning algorithms.

12 pages, 2 figures

References in corpus (1)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks

Cited by in corpus (12)

• Associative Memories via Predictive Coding
• AdaSwarm: Augmenting Gradient-Based optimizers in Deep Learning with Swarm Intelligence
• The Quantum Path Kernel: a Generalized Quantum Neural Tangent Kernel for Deep Quantum Machine Learning
• Computing on Functions Using Randomized Vector Representations
• Data-driven geophysical forecasting: Simple, low-cost, and accurate baselines with kernel methods
• Online Limited Memory Neural-Linear Bandits with Likelihood Matching
• Learning curves for Gaussian process regression with power-law priors and targets
• CETransformer: Casual Effect Estimation via Transformer Based Representation Learning
• Neural Networks as Kernel Learners: The Silent Alignment Effect
• New Insights into Graph Convolutional Networks using Neural Tangent Kernels
• Gradient Kernel Regression
• Self-supervised denoising for massive noisy images