Spectrum concentration in deep residual learning: a free probability approach

Abstract

We revisit the initialization of deep residual networks (ResNets) by introducing a novel analytical tool in free probability to the community of deep learning. This tool deals with non-Hermitian random matrices, rather than their conventional Hermitian counterparts in the literature. As a consequence, this new tool enables us to evaluate the singular value spectrum of the input-output Jacobian of a fully-connected deep ResNet for both linear and nonlinear cases. With the powerful tool of free probability, we conduct an asymptotic analysis of the spectrum on the single-layer case, and then extend this analysis to the multi-layer case of an arbitrary number of layers. In particular, we propose to rescale the classical random initialization by the number of residual units, so that the spectrum has the order of $O(1)$, when compared with the large width and depth of the network. We empirically demonstrate that the proposed initialization scheme learns at a speed of orders of magnitudes faster than the classical ones, and thus attests a strong practical relevance of this investigation.

References in corpus (7)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Universal and non-universal properties of cross-correlations in financial time series
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Products of Rectangular Random Matrices: Singular Values and Progressive Scattering
• The Emergence of Spectral Universality in Deep Networks
• Identity Matters in Deep Learning
• Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function

Cited by in corpus (2)

• A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks
• Tensor Programs III: Neural Matrix Laws