PaReprop: Fast Parallelized Reversible Backpropagation

Tyler Zhu, Karttikeya Mangalam

The growing size of datasets and deep learning models has made faster and memory-efficient training crucial. Reversible transformers have recently been introduced as an exciting ne…