Beyond the Mean: Fisher-Orthogonal Projection for Natural Gradient Descent in Large Batch Training

Yishun Lu, Wesley Armour

Modern GPUs are equipped with large amounts of high-bandwidth memory, enabling them to support mini-batch sizes of up to tens of thousands of training samples. However, most existi…