Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes

Abstract

There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs). This equivalence enables, for instance, test set predictions that would have resulted from a fully Bayesian, infinitely wide trained FCN to be computed without ever instantiating the FCN, but by instead evaluating the corresponding GP. In this work, we derive an analogous equivalence for multi-layer convolutional neural networks (CNNs) both with and without pooling layers, and achieve state of the art results on CIFAR10 for GPs without trainable kernels. We also introduce a Monte Carlo method to estimate the GP corresponding to a given neural network architecture, even in cases where the analytic form has too many terms to be computationally feasible. Surprisingly, in the absence of pooling layers, the GPs corresponding to CNNs with and without weight sharing are identical. As a consequence, translation equivariance, beneficial in finite channel CNNs trained with stochastic gradient descent (SGD), is guaranteed to play no role in the Bayesian treatment of the infinite channel limit - a qualitative difference between the two regimes that is not present in the FCN case. We confirm experimentally, that while in some scenarios the performance of SGD-trained finite CNNs approaches that of the corresponding GPs as the channel count increases, with careful tuning SGD-trained CNNs can significantly outperform their corresponding GPs, suggesting advantages from SGD training compared to fully Bayesian parameter estimation.

Published as a conference paper at ICLR 2019

Cited by in corpus (97)

• On Exact Computation with an Infinitely Wide Neural Net
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Explaining Neural Scaling Laws
• Optimization for deep learning: theory and algorithms
• Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
• When Do Neural Networks Outperform Kernel Methods?
• The large learning rate phase of deep learning: the catapult mechanism
• Neural Tangents: Fast and Easy Infinite Neural Networks in Python
• A Mean Field Theory of Batch Normalization
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Asymptotics of Wide Networks from Feynman Diagrams
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• Quantum field-theoretic machine learning
• Finite Versus Infinite Neural Networks: an Empirical Study
• A Fine-Grained Spectral Perspective on Neural Networks
• Nonperturbative renormalization for the neural network-QFT correspondence
• A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• Neural Kernels Without Tangents
• The Functional Neural Process
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Frequency Bias in Neural Networks for Input of Non-Uniform Density
• On the training dynamics of deep networks with $L_2$ regularization
• Asymptotics of representation learning in finite Bayesian neural networks
• Bayesian Neural Network Priors Revisited
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Finite size corrections for neural network Gaussian processes
• Tensor Programs III: Neural Matrix Laws
• Quantum neural networks form Gaussian processes
• Locality defeats the curse of dimensionality in convolutional teacher-student scenarios
• Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics
• Towards NNGP-guided Neural Architecture Search
• Asymptotics of Wide Convolutional Neural Networks
• Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
• Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks
• Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networks
• Temperature check: theory and practice for training models with softmax-cross-entropy losses
• Non-Gaussian processes and neural networks at finite widths
• Approximate Inference Turns Deep Networks into Gaussian Processes
• Generalization bounds for deep learning
• On the Similarity between the Laplace and Neural Tangent Kernels
• Towards Learning Convolutions from Scratch
• Computational Separation Between Convolutional and Fully-Connected Networks
• Neural networks are a priori biased towards Boolean functions with low entropy
• Revisiting Spatial Invariance with Low-Rank Local Connectivity
• Infinitely Wide Graph Convolutional Networks: Semi-supervised Learning via Gaussian Processes
• Average-case Acceleration Through Spectral Density Estimation
• On Infinite-Width Hypernetworks
• Conditional Temporal Neural Processes with Covariance Loss
• Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
• Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias
• Benign Overfitting and Noisy Features
• Wide Neural Networks with Bottlenecks are Deep Gaussian Processes
• The Ridgelet Prior: A Covariance Function Approach to Prior Specification for Bayesian Neural Networks
• Exact posterior distributions of wide Bayesian neural networks
• Dataset Meta-Learning from Kernel Ridge-Regression
• The Recurrent Neural Tangent Kernel
• Richer priors for infinitely wide multi-layer perceptrons
• Precise characterization of the prior predictive distribution of deep ReLU networks
• The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
• Approximation and Learning with Deep Convolutional Models: a Kernel Perspective
• Why bigger is not always better: on finite and infinite neural networks
• Non-asymptotic approximations of neural networks by Gaussian processes
• Meta-Learning with Neural Tangent Kernels
• Correlated Weights in Infinite Limits of Deep Convolutional Neural Networks
• Scalable Training of Inference Networks for Gaussian-Process Models
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• Infinitely Wide Tensor Networks as Gaussian Process
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• Scale Mixtures of Neural Network Gaussian Processes
• Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
• Why flatness does and does not correlate with generalization for deep neural networks
• Deep kernel processes
• Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity
• Infinite-channel deep stable convolutional neural networks
• Adversarial Robustness Guarantees for Random Deep Neural Networks
• A Probabilistic Representation of Deep Learning for Improving The Information Theoretic Interpretability
• Scaling Neural Tangent Kernels via Sketching and Random Features
• Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?
• On the validity of kernel approximations for orthogonally-initialized neural networks
• Implicit Bias of Linear RNNs
• A Temporal Kernel Approach for Deep Learning with Continuous-time Information
• The role of a layer in deep neural networks: a Gaussian Process perspective
• Deep Neural Networks as Point Estimates for Deep Gaussian Processes
• Explicitly Bayesian Regularizations in Deep Learning
• Uncertainty-aware Cardinality Estimation by Neural Network Gaussian Process
• Learning curves for Gaussian process regression with power-law priors and targets
• Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping
• A variational approximate posterior for the deep Wishart process
• Enhanced Recurrent Neural Tangent Kernels for Non-Time-Series Data
• Faster Kernel Interpolation for Gaussian Processes
• Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks
• Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks
• Large-Scale Learning with Fourier Features and Tensor Decompositions
• Learning with Neural Tangent Kernels in Near Input Sparsity Time
• Disentangling deep neural networks with rectified linear units using duality