Directional Analysis of Stochastic Gradient Descent via von Mises-Fisher Distributions in Deep learning

Cheolhyoung Lee, Kyunghyun Cho, Wanmo Kang

Although stochastic gradient descent (SGD) is a driving force behind the recent success of deep learning, our understanding of its dynamics in a high-dimensional parameter space is…