Accelerating Neural Network Training Along Sharp and Flat Directions

Daniyar Zakarin, Sidak Pal Singh

Recent work has highlighted a surprising alignment between gradients and the top eigenspace of the Hessian -- termed the Dominant subspace -- during neural network training. Concur…