Asymptotics of Wide Convolutional Neural Networks

Abstract

Wide neural networks have proven to be a rich class of architectures for both theory and practice. Motivated by the observation that finite width convolutional networks appear to outperform infinite width networks, we study scaling laws for wide CNNs and networks with skip connections. Following the approach of (Dyer & Gur-Ari, 2019), we present a simple diagrammatic recipe to derive the asymptotic width dependence for many quantities of interest. These scaling relationships provide a solvable description for the training dynamics of wide convolutional networks. We test these relations across a broad range of architectures. In particular, we find that the difference in performance between finite and infinite width models vanishes at a definite rate with respect to model width. Nonetheless, this relation is consistent with finite width models generalizing either better or worse than their infinite width counterparts, and we provide examples where the relative performance depends on the optimization details.

arXiv comment23 pages, 12 figures

References in corpus (15)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Scaling description of generalization with number of parameters in deep learning
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Enhanced Convolutional Neural Tangent Kernels
• SGD Learns the Conjugate Kernel Class of the Network
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• Non-Gaussian processes and neural networks at finite widths
• On the asymptotics of wide networks with polynomial activations

Cited by in corpus (7)

• Explaining Neural Scaling Laws
• Finite Versus Infinite Neural Networks: an Empirical Study
• Asymptotics of representation learning in finite Bayesian neural networks
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• Non-asymptotic approximations of neural networks by Gaussian processes
• Neural Networks as Kernel Learners: The Silent Alignment Effect