AOS: Adaptive Optimizer Switching via Training-State Signals for Faster Convergence and Better Generalization

Alok Kumar Pandey, Umang Chaturvedi, Aatish Rana +1

Single-optimizer training is a poor fit for the distinct phases of deep network optimization: adaptive methods handle noisy early gradients well but overshoot flat minima, while SG…