Effective continuous equations for adaptive SGD: a stochastic analysis view

Luca Callisti, Marco Romito, Francesco Triggiano

We present a theoretical analysis of some popular adaptive Stochastic Gradient Descent (SGD) methods in the small learning rate regime. Using the stochastic modified equations fram…