Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective

Abstract

In the past decade, deep neural networks (DNNs) came to the fore as the leading machine learning algorithms for a variety of tasks. Their raise was founded on market needs and engineering craftsmanship, the latter based more on trial and error than on theory. While still far behind the application forefront, the theoretical study of DNNs has recently made important advancements in analyzing the highly over-parameterized regime where some exact results have been obtained. Leveraging these ideas and adopting a more physics-like approach, here we construct a versatile field-theory formalism for supervised deep learning, involving renormalization group, Feynman diagrams and replicas. In particular we show that our approach leads to highly accurate predictions of learning curves of truly deep DNNs trained on polynomial regression tasks and that these predictions can be used for efficient hyper-parameter optimization. In addition, they explain how DNNs generalize well despite being highly over-parameterized, this due to an entropic bias to simple functions which, for the case of fully-connected DNNs with data sampled on the hypersphere, are low order polynomials in the input vector. Being a complex interacting system of artificial neurons, we believe that such tools and methodologies borrowed from condensed matter physics would prove essential for obtaining an accurate quantitative understanding of deep learning.

References in corpus (6)

• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Quantum Entanglement in Deep Learning Architectures
• Periodic training of creeping solids
• Geometry of energy landscapes and the optimizability of deep neural networks
• Exploring the Function Space of Deep-Learning Machines
• A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines

Cited by in corpus (10)

• Neural Networks and Quantum Field Theory
• Random matrix analysis of deep neural network weight matrices
• Introduction to dynamical mean-field theory of randomly connected neural networks with bidirectionally correlated couplings
• Deep neural networks have an inbuilt Occam's razor
• Decomposing neural networks as mappings of correlation functions
• Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft Committee Machines
• Boundary between noise and information applied to filtering neural network weight matrices
• Bayesian RG Flow in Neural Network Field Theories
• Wilsonian Renormalization of Neural Network Gaussian Processes
• Predicting the outputs of finite deep neural networks trained with noisy gradients