Robust Learning Rate Selection for Stochastic Optimization via Splitting Diagnostic

Matteo Sordello, Niccolò Dalmasso, Hangfeng He +1

This paper proposes SplitSGD, a new dynamic learning rate schedule for stochastic optimization. This method decreases the learning rate for better adaptation to the local geometry…