When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay

Hasan Amin, Wei-Kai Chang, Rajiv Khanna

Normalization renders large parts of neural networks effectively scale invariant, inducing a hidden feedback loop in which learning-rate schedules and weight decay interact through…