Beyond a Single Explanation of the Adam--SGD Gap

Chenxiang Zhang, Rustem Islamov, Enea Monzio Compagnoni +3

Prior work has identified several factors that can contribute to the performance gap between Adam and SGD, spanning data aspects, architecture design, and optimization properties.…