Geometry Perspective Of Estimating Learning Capability Of Neural Networks

Abstract

The paper uses statistical and differential geometric motivation to acquire prior information about the learning capability of an artificial neural network on a given dataset. The paper considers a broad class of neural networks with generalized architecture performing simple least square regression with stochastic gradient descent (SGD). The system characteristics at two critical epochs in the learning trajectory are analyzed. During some epochs of the training phase, the system reaches equilibrium with the generalization capability attaining a maximum. The system can also be coherent with localized, non-equilibrium states, which is characterized by the stabilization of the Hessian matrix. The paper proves that neural networks with higher generalization capability will have a slower convergence rate. The relationship between the generalization capability with the stability of the neural network has also been discussed. By correlating the principles of high-energy physics with the learning theory of neural networks, the paper establishes a variant of the Complexity-Action conjecture from an artificial neural network perspective.

arXiv comment15 pages

References in corpus (7)

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Complexity Equals Action
• Opening the Black Box of Deep Neural Networks via Information
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Sharp Minima Can Generalize For Deep Nets
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• An analytic theory of generalization dynamics and transfer learning in deep linear networks