Enhanced Convolutional Neural Tangent Kernels

Abstract

Recent research shows that for training with ${\ell }_2$ loss, convolutional neural networks (CNNs) whose width (number of channels in convolutional layers) goes to infinity correspond to regression with respect to the CNN Gaussian Process kernel (CNN-GP) if only the last layer is trained, and correspond to regression with respect to the Convolutional Neural Tangent Kernel (CNTK) if all layers are trained. An exact algorithm to compute CNTK (Arora et al., 2019) yielded the finding that classification accuracy of CNTK on CIFAR-10 is within 6-7% of that of that of the corresponding CNN architecture (best figure being around 78%) which is interesting performance for a fixed kernel. Here we show how to significantly enhance the performance of these kernels using two ideas. (1) Modifying the kernel using a new operation called Local Average Pooling (LAP) which preserves efficient computability of the kernel and inherits the spirit of standard data augmentation using pixel shifts. Earlier papers were unable to incorporate naive data augmentation because of the quadratic training cost of kernel regression. This idea is inspired by Global Average Pooling (GAP), which we show for CNN-GP and CNTK is equivalent to full translation data augmentation. (2) Representing the input image using a pre-processing technique proposed by Coates et al. (2011), which uses a single convolutional layer composed of random image patches. On CIFAR-10, the resulting kernel, CNN-GP with LAP and horizontal flip data augmentation, achieves 89% accuracy, matching the performance of AlexNet (Krizhevsky et al., 2012). Note that this is the best such result we know of for a classifier that is not a trained neural network. Similar improvements are obtained for Fashion-MNIST.

References in corpus (7)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Striving for Simplicity: The All Convolutional Net
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Do ImageNet Classifiers Generalize to ImageNet?
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation

Cited by in corpus (41)

• When Do Neural Networks Outperform Kernel Methods?
• Differentially Private Learning Needs Better Features (or Much More Data)
• Finite Versus Infinite Neural Networks: an Empirical Study
• Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed
• Learning with invariances in random features and kernel models
• Kernel Alignment Risk Estimator: Risk Prediction from Training Data
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
• Towards NNGP-guided Neural Architecture Search
• Asymptotics of Wide Convolutional Neural Networks
• Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width
• The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks
• On the Similarity between the Laplace and Neural Tangent Kernels
• Rethinking Influence Functions of Neural Networks in the Over-parameterized Regime
• On Infinite-Width Hypernetworks
• Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK
• Shift Invariance Can Reduce Adversarial Robustness
• Why bigger is not always better: on finite and infinite neural networks
• Generalization Guarantees for Neural Architecture Search with Train-Validation Split
• Approximation and Learning with Deep Convolutional Models: a Kernel Perspective
• Neural Tangent Kernel Maximum Mean Discrepancy
• Finding trainable sparse networks through Neural Tangent Transfer
• An Optimization and Generalization Analysis for Max-Pooling Networks
• Global Convergence and Generalization Bound of Gradient-Based Meta-Learning with Deep Neural Nets
• Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
• Learning with convolution and pooling operations in kernel methods
• Implicit bias of deep linear networks in the large learning rate phase
• Global Convergence of SGD On Two Layer Neural Nets
• The Three Stages of Learning Dynamics in High-Dimensional Kernel Methods
• Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity
• The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods
• Collegial Ensembles
• Adversarial Robustness Guarantees for Random Deep Neural Networks
• Fast Adaptation with Linearized Neural Networks
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks
• Deep Networks Provably Classify Data on Curves
• Largest Eigenvalues of the Conjugate Kernel of Single-Layered Neural Networks
• Quantifying Epistemic Uncertainty in Deep Learning
• Disentangling deep neural networks with rectified linear units using duality
• A Revision of Neural Tangent Kernel-based Approaches for Neural Networks
• Enhanced Recurrent Neural Tangent Kernels for Non-Time-Series Data