Inefficiency of K-FAC for Large Batch Size Training

Linjian Ma, Gabe Montague, Jiayu Ye +4

In stochastic optimization, using large batch sizes during training can leverage parallel resources to produce faster wall-clock training times per training epoch. However, for bot…