Curvature-Weighted Gradient Diversity: A Noise Measure for Geometry-Adaptive SGD Schedules

Muhammad Hamza, Ayush Goel

The standard convergence analysis of mini-batch stochastic gradient descent (SGD) models gradient noise using a single variance term that treats all parameter directions equally, i…