Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants

Depen Morwani, Nikhil Vyas, Hanlin Zhang +1

Recent advancements in deep learning optimization have introduced new algorithms, such as Schedule-Free optimizers, AdEMAMix, MARS and Lion which modify traditional momentum mechan…