Wide Neural Networks with Bottlenecks are Deep Gaussian Processes
arXiv:2001.00921
Abstract
There has recently been much work on the "wide limit" of neural networks, where Bayesian neural networks (BNNs) are shown to converge to a Gaussian process (GP) as all hidden layers are sent to infinite width. However, these results do not apply to architectures that require one or more of the hidden layers to remain narrow. In this paper, we consider the wide limit of BNNs where some hidden layers, called "bottlenecks", are held at finite width. The result is a composition of GPs that we term a "bottleneck neural network Gaussian process" (bottleneck NNGP). Although intuitive, the subtlety of the proof is in showing that the wide limit of a composition of networks is in fact the composition of the limiting GPs. We also analyze theoretically a single-bottleneck NNGP, finding that the bottleneck induces dependence between the outputs of a multi-output network that persists through extreme post-bottleneck depths, and prevents the kernel of the network from losing discriminative power at extreme post-bottleneck depths.
References in corpus (20)
- Adam: A Method for Stochastic Optimization
- Distributed Representations of Words and Phrases and their Compositionality
- Auto-Encoding Variational Bayes
- An Overview of Multi-Task Learning in Deep Neural Networks
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Deep Neural Networks as Gaussian Processes
- Variational Dropout and the Local Reparameterization Trick
- On Exact Computation with an Infinitely Wide Neural Net
- Exponential expressivity in deep neural networks through transient chaos
- Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
- Deep Gaussian Processes for Regression using Approximate Expectation Propagation
- Random Feature Expansions for Deep Gaussian Processes
- Avoiding pathologies in very deep networks
- Variational Auto-encoded Deep Gaussian Processes
- Steps Toward Deep Kernel Methods from Infinite Neural Networks
- Nested Variational Compression in Deep Gaussian Processes
- Gaussian Process Conditional Density Estimation
- How far can we go without convolution: Improving fully-connected networks