Understanding the Training Speedup from Sampling with Approximate Losses

Rudrajit Das, Xi Chen, Bertram Ieong +2

It is well known that selecting samples with large losses/gradients can significantly reduce the number of training steps. However, the selection overhead is often too high to yiel…