Optimal Condition for Initialization Variance in Deep Neural Networks: An SGD Dynamics Perspective

Hiroshi Horii, Sothea Has

Stochastic gradient descent (SGD), one of the most fundamental optimization algorithms in machine learning (ML), can be recast through a continuous-time approximation as a Fokker-P…