Deep Convolutional Networks as shallow Gaussian Processes

Abstract

We show that the output of a (residual) convolutional neural network (CNN) with an appropriate prior over the weights and biases is a Gaussian process (GP) in the limit of infinitely many convolutional filters, extending similar results for dense networks. For a CNN, the equivalent kernel can be computed exactly and, unlike "deep kernels", has very few parameters: only the hyperparameters of the original CNN. Further, we show that this kernel has two properties that allow it to be computed efficiently; the cost of evaluating the kernel for a pair of images is similar to a single forward pass through the original CNN with only one filter per layer. The kernel equivalent to a 32-layer ResNet obtains 0.84% classification error on MNIST, a new record for GPs with a comparable number of parameters.

References in corpus (9)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Intriguing properties of neural networks
• Wide Residual Networks
• Neural Ordinary Differential Equations
• Deep Neural Networks as Gaussian Processes
• Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks
• Convolutional Gaussian Processes
• Deep Gaussian Processes with Convolutional Kernels
• A Gaussian Process perspective on Convolutional Neural Networks

Cited by in corpus (38)

• The generalization error of random features regression: Precise asymptotics and double descent curve
• Neural Operator: Graph Kernel Network for Partial Differential Equations
• Explaining Neural Scaling Laws
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Neural Networks and Quantum Field Theory
• A Mean Field Theory of Batch Normalization
• Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• Quantum field-theoretic machine learning
• A Fine-Grained Spectral Perspective on Neural Networks
• Nonperturbative renormalization for the neural network-QFT correspondence
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspective
• Finite size corrections for neural network Gaussian processes
• Generalization bounds for deep learning
• Neural networks are a priori biased towards Boolean functions with low entropy
• Bayesian Deep Learning via Subnetwork Inference
• On the expected behaviour of noise regularised deep neural networks as Gaussian processes
• A Bayesian Perspective on the Deep Image Prior
• Infinite Neural Network Quantum States: Entanglement and Training Dynamics
• Conditional Temporal Neural Processes with Covariance Loss
• Benign Overfitting and Noisy Features
• Linearly Constrained Gaussian Processes with Boundary Conditions
• Richer priors for infinitely wide multi-layer perceptrons
• Infinitely Wide Tensor Networks as Gaussian Process
• Reducing the Amortization Gap in Variational Autoencoders: A Bayesian Random Function Approach
• Deep kernel processes
• Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity
• Posterior contraction rates for constrained deep Gaussian processes in density estimation and classication
• On Neural Networks as Infinite Tree-Structured Probabilistic Graphical Models
• Uncertainty-aware Cardinality Estimation by Neural Network Gaussian Process
• Explicitly Bayesian Regularizations in Deep Learning
• Variational Policy Search using Sparse Gaussian Process Priors for Learning Multimodal Optimal Actions
• A variational approximate posterior for the deep Wishart process
• Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks
• Random matrices in service of ML footprint: ternary random features with no performance loss
• Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping