Reducing Per-Sample Interference in Stochastic Optimization

Apostolos Avranas

Modern optimizers combine gradients from the current mini-batch with historical optimization state, such as momentum or adaptive moments. While effective, this standard practice ca…