Accelerating Large Batch Training via Gradient Signal to Noise Ratio (GSNR)

Guo-qing Jiang, Jinlong Liu, Zixiang Ding +2

As models for nature language processing (NLP), computer vision (CV) and recommendation systems (RS) require surging computation, a large number of GPUs/TPUs are paralleled as a la…