On the expected behaviour of noise regularised deep neural networks as Gaussian processes

Abstract

Recent work has established the equivalence between deep neural networks and Gaussian processes (GPs), resulting in so-called neural network Gaussian processes (NNGPs). The behaviour of these models depends on the initialisation of the corresponding network. In this work, we consider the impact of noise regularisation (e.g. dropout) on NNGPs, and relate their behaviour to signal propagation theory in noise regularised deep neural networks. For ReLU activations, we find that the best performing NNGPs have kernel parameters that correspond to a recently proposed initialisation scheme for noise regularised ReLU networks. In addition, we show how the noise influences the covariance matrix of the NNGP, producing a stronger prior towards simple functions away from the training points. We verify our theoretical findings with experiments on MNIST and CIFAR-10 as well as on synthetic data.

arXiv comment8 pages, 6 figures, preliminary work

References in corpus (17)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Deep Neural Networks as Gaussian Processes
• Artificial Intelligence and Statistics
• Exponential expressivity in deep neural networks through transient chaos
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Stochastic Variational Deep Kernel Learning
• Deep Convolutional Networks as shallow Gaussian Processes
• Gaussian Process Behaviour in Wide Deep Neural Networks
• Avoiding pathologies in very deep networks
• Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
• Variational Auto-encoded Deep Gaussian Processes
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• Nested Variational Compression in Deep Gaussian Processes
• Mean Field Residual Networks: On the Edge of Chaos
• Deep Information Propagation
• Critical initialisation for deep signal propagation in noisy rectifier neural networks

Cited by in corpus (3)

• Finite Versus Infinite Neural Networks: an Empirical Study
• Towards NNGP-guided Neural Architecture Search
• Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit