Neural Kernels Without Tangents

Abstract

We investigate the connections between neural networks and simple building blocks in kernel space. In particular, using well established feature space tools such as direct sum, averaging, and moment lifting, we present an algebra for creating "compositional" kernels from bags of features. We show that these operations correspond to many of the building blocks of "neural tangent kernels (NTK)". Experimentally, we show that there is a correlation in test error between neural network architectures and the associated kernels. We construct a simple neural network architecture using only 3x3 convolutions, 2x2 average pooling, ReLU, and optimized with SGD and MSE loss that achieves 96% accuracy on CIFAR10, and whose corresponding compositional kernel achieves 90% accuracy. We also use our constructions to investigate the relative performance of neural networks, NTKs, and compositional kernels in the small dataset regime. In particular, we find that compositional kernels outperform NTKs and neural networks outperform both kernel methods.

code used to produce our results can be found at: https://github.com/modestyachts/neural_kernels_code

References in corpus (7)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Convolutional Kernel Networks
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• A Kernel Theory of Modern Data Augmentation

Cited by in corpus (25)

• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• Explaining Neural Scaling Laws
• Self-Distillation Amplifies Regularization in Hilbert Space
• When Do Neural Networks Outperform Kernel Methods?
• Differentially Private Learning Needs Better Features (or Much More Data)
• Finite Versus Infinite Neural Networks: an Empirical Study
• Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Kernel Alignment Risk Estimator: Risk Prediction from Training Data
• Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Towards NNGP-guided Neural Architecture Search
• Distributional Generalization: A New Kind of Generalization
• Convex Geometry and Duality of Over-parameterized Neural Networks
• The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
• Approximation and Learning with Deep Convolutional Models: a Kernel Perspective
• Generalization Guarantees for Neural Architecture Search with Train-Validation Split
• Correlated Weights in Infinite Limits of Deep Convolutional Neural Networks
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
• Detection of asteroid trails in Hubble Space Telescope images using Deep Learning
• The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods
• How rotational invariance of common kernels prevents generalization in high dimensions
• VC dimension of partially quantized neural networks in the overparametrized regime
• Neural Optimization Kernel: Towards Robust Deep Learning