#generalization

Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate

Zhiyuan Li, Kaifeng Lyu, Sanjeev Arora

The paper proposes an 'intrinsic learning rate' that combines the usual learning rate and weight decay to model the training dynamics of normalized deep networks via stochastic dif…