Nonlinear gradient mappings and stochastic optimization: A general framework with applications to heavy-tail noise

Dusan Jakovetic, Dragana Bajovic, Anit Kumar Sahu +3

We introduce a general framework for nonlinear stochastic gradient descent (SGD) for the scenarios when gradient noise exhibits heavy tails. The proposed framework subsumes several…