Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients?

Zitao Song, Cedar Site Bai, Zhe Zhang +2

Training instabilities such as loss spikes are frequently the result of stochastic gradient noise. Because of rare expressions in language training data, and multiple layer composi…