What do near-optimal learning rate schedules look like?

Hiroki Naganuma, Atish Agarwala, Priya Kasimbeg +1

A basic unanswered question in neural network training is: what is the best learning rate schedule shape for a given workload? The choice of learning rate schedule is a key factor…