Fluctuation-dissipation Type Theorem in Stochastic Linear Learning

Abstract

The fluctuation-dissipation theorem (FDT) is a simple yet powerful consequence of the first-order differential equation governing the dynamics of systems subject simultaneously to dissipative and stochastic forces. The linear learning dynamics, in which the input vector maps to the output vector by a linear matrix whose elements are the subject of learning, has a stochastic version closely mimicking the Langevin dynamics when a full-batch gradient descent scheme is replaced by that of stochastic gradient descent. We derive a generalized FDT for the stochastic linear learning dynamics and verify its validity among the well-known machine learning data sets such as MNIST, CIFAR-10 and EMNIST.

References in corpus (10)

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• EMNIST: an extension of MNIST to handwritten letters
• Stochastic Dynamical Structure (SDS) of Nonequilibrium Processes in the Absence of Detailed Balance. II: construction of SDS with nonlinear force and multiplicative noise
• A mathematical theory of semantic development in deep neural networks
• Structure of Stochastic Dynamics near Fixed Points
• Stochastic Dynamical Structure (SDS) of Nonequilibrium Processes in the Absence of Detailed Balance. III: potential function in local stochastic dynamics and in steady state of Boltzmann-Gibbs type distribution function
• Stochastic thermodynamics for delayed Langevin systems
• Non-equilibrium fluctuations for linear diffusion dynamics
• Fluctuation-dissipation relations for stochastic gradient descent
• Nonequilibrium Steady State Driven by a Nonlinear Drift Force

Cited by in corpus (4)

• Stochastic Gradient Descent-like relaxation is equivalent to Metropolis dynamics in discrete optimization and inference problems
• Macroscopic fluctuation-response theory and its use for gene regulatory networks
• Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation
• Stochastic Gradient Descent and Anomaly of Variance-flatness Relation in Artificial Neural Networks