Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks

Abstract

Analysing and computing with Gaussian processes arising from infinitely wide neural networks has recently seen a resurgence in popularity. Despite this, many explicit covariance functions of networks with activation functions used in modern networks remain unknown. Furthermore, while the kernels of deep networks can be computed iteratively, theoretical understanding of deep kernels is lacking, particularly with respect to fixed-point dynamics. Firstly, we derive the covariance functions of multi-layer perceptrons (MLPs) with exponential linear units (ELU) and Gaussian error linear units (GELU) and evaluate the performance of the limiting Gaussian processes on some benchmarks. Secondly, and more generally, we analyse the fixed-point dynamics of iterated kernels corresponding to a broad range of activation functions. We find that unlike some previously studied neural network kernels, these new kernels exhibit non-trivial fixed-point dynamics which are mirrored in finite-width neural networks. The fixed point behaviour present in some networks explains a mechanism for implicit regularisation in overparameterised deep models. Our results relate to both the static iid parameter conjugate kernel and the dynamic neural tangent kernel constructions. Software at github.com/RussellTsuchida/ELU_GELU_kernels.

AAAI camera ready version. 18 pages, 9 figures, 2 tables. Corrected name particle capitalisation and formatting

References in corpus (18)

• Gaussian Error Linear Units (GELUs)
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Searching for Activation Functions
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Deep Neural Networks as Gaussian Processes
• On Exact Computation with an Infinitely Wide Neural Net
• Exponential expressivity in deep neural networks through transient chaos
• On Lazy Training in Differentiable Programming
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Deep Convolutional Networks as shallow Gaussian Processes
• A Fine-Grained Spectral Perspective on Neural Networks
• Exact expressions for double descent and implicit regularization via surrogate random design
• Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
• Stable behaviour of infinitely wide deep neural networks
• Richer priors for infinitely wide multi-layer perceptrons