The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective

Abstract

Large width limits have been a recent focus of deep learning research: modulo computational practicalities, do wider networks outperform narrower ones? Answering this question has been challenging, as conventional networks gain representational power with width, potentially masking any negative effects. Our analysis in this paper decouples capacity and width via the generalization of neural networks to Deep Gaussian Processes (Deep GP), a class of nonparametric hierarchical models that subsume neural nets. In doing so, we aim to understand how width affects (standard) neural networks once they have sufficient capacity for a given modeling task. Our theoretical and empirical results on Deep GP suggest that large width can be detrimental to hierarchical models. Surprisingly, we prove that even nonparametric Deep GP converge to Gaussian processes, effectively becoming shallower without any increase in representational power. The posterior, which corresponds to a mixture of data-adaptable basis functions, becomes less data-dependent with width. Our tail analysis demonstrates that width and depth have opposite effects: depth accentuates a model's non-Gaussianity, while width makes models increasingly Gaussian. We find there is a "sweet spot" that maximizes test performance before the limiting GP behavior prevents adaptability, occurring at width = 1 or width = 2 for nonparametric Deep GP. These results make strong predictions about the same phenomenon in conventional neural networks trained with L2 regularization (analogous to a Gaussian prior on parameters): we show that such neural networks may need up to 500 - 1000 hidden units for sufficient capacity - depending on the dataset - but further width degrades performance.

arXiv commentNeurIPS 2021

References in corpus (21)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• On the Number of Linear Regions of Deep Neural Networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Scaling description of generalization with number of parameters in deep learning
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Random Feature Expansions for Deep Gaussian Processes
• Avoiding pathologies in very deep networks
• Neural Networks and Quantum Field Theory
• Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Understanding Priors in Bayesian Neural Networks at the Unit Level
• Neural Kernels Without Tangents
• Infinite attention: NNGP and NTK for deep attention networks
• Interpretable deep Gaussian processes with moments
• Exact marginal prior distributions of finite Bayesian neural networks
• GPflux: A Library for Deep Gaussian Processes
• Exact posterior distributions of wide Bayesian neural networks

Cited by in corpus (3)

• Exact marginal prior distributions of finite Bayesian neural networks
• Dependence between Bayesian neural network units
• Conditional Deep Gaussian Processes: empirical Bayes hyperdata learning