Phase Diagram of Initial Condensation for Two-layer Neural Networks

Abstract

The phenomenon of distinct behaviors exhibited by neural networks under varying scales of initialization remains an enigma in deep learning research. In this paper, based on the earlier work by Luo et al.~\cite{luo2021phase}, we present a phase diagram of initial condensation for two-layer neural networks. Condensation is a phenomenon wherein the weight vectors of neural networks concentrate on isolated orientations during the training process, and it is a feature in non-linear learning process that enables neural networks to possess better generalization abilities. Our phase diagram serves to provide a comprehensive understanding of the dynamical regimes of neural networks and their dependence on the choice of hyperparameters related to initialization. Furthermore, we demonstrate in detail the underlying mechanisms by which small initialization leads to condensation at the initial training stage.

References in corpus (9)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Gradient Descent Finds Global Minima of Deep Neural Networks
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• Phase diagram for two-layer ReLU neural networks at infinite-width limit
• Towards Understanding the Condensation of Neural Networks at Initial Training
• Towards an Understanding of Residual Networks Using Neural Tangent Hierarchy (NTH)
• Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width