ROAM: memory-efficient large DNN training via optimized operator ordering and memory layout

Huiyao Shu, Ang Wang, Ziji Shi +3

As deep learning models continue to increase in size, the memory requirements for training have surged. While high-level techniques like offloading, recomputation, and compression…