State-Aware Activation Recomputation with Replay Contracts and Selective Preservation

Annan Zhang, Shengjie Song, Wenshuai Yao +2

Activation recomputation saves accelerator memory by replaying forward segments during the backward pass, but replay is only correct if it sees the state the original forward pass…