Large Learning Rates Improve Generalization: But How Large Are We Talking About?

Ekaterina Lobacheva, Eduard Pockonechnyy, Maxim Kodryan +1

Inspired by recent research that recommends starting neural networks training with large learning rates (LRs) to achieve the best generalization, we explore this hypothesis in deta…