Control of Overfitting with Physics

Abstract

While there are many works on the applications of machine learning, not so many of them are trying to understand the theoretical justifications to explain their efficiency. In this work, overfitting control (or generalization property) in machine learning is explained using analogies from physics and biology. For stochastic gradient Langevin dynamics, we show that the Eyring formula of kinetic theory allows to control overfitting in the algorithmic stability approach - when wide minima of the risk function with low free energy correspond to low overfitting. For the generative adversarial network (GAN) model, we establish an analogy between GAN and the predator-prey model in biology. An application of this analogy allows us to explain the selection of wide likelihood maxima and overfitting reduction for GANs.

20 pages, 6 figures

References in corpus (8)

• Quantum reinforcement learning
• Teaching the Environment to Control Quantum Systems
• Gradient descent GAN optimization is locally stable
• Almost-everywhere algorithmic stability and generalization error
• The Arrhenius formula in kinetic theory and Witten's spectral asymptotics
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations
• Lotka-Volterra Model with Mutations and Generative Adversarial Networks
• Functional Space Analysis of Local GAN Convergence