Layer-wise Adaptive Step-Sizes for Stochastic First-Order Methods for Deep Learning

Achraf Bahamou, Donald Goldfarb

We propose a new per-layer adaptive step-size procedure for stochastic first-order optimization methods for minimizing empirical loss functions in deep learning, eliminating the ne…