2BP: 2-Stage Backpropagation

Christopher Rae, Joseph K. L. Lee, James Richings

As Deep Neural Networks (DNNs) grow in size and complexity, they often exceed the memory capacity of a single accelerator, necessitating the sharding of model parameters across mul…