Mean-field analysis for heavy ball methods: Dropout-stability, connectivity, and global convergence

Diyuan Wu, Vyacheslav Kungurtsev, Marco Mondelli

The stochastic heavy ball method (SHB), also known as stochastic gradient descent (SGD) with Polyak's momentum, is widely used in training neural networks. However, despite the rem…