67 citations · 82 across the 8 of their papers we have counts for
Showing 2022Show all
2 papers · 1 filter
cs.LG2022★ 67 cited
Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models
Zhiquan Lai, Shengwei Li, Xudong Tang +5
Foundation models are becoming the dominant deep learning technologies. Pretraining a foundation model is always time-consumed due to the large scale of both the model parameter an…
cs.LG2022★ 6 cited
DELTA: Dynamically Optimizing GPU Memory beyond Tensor Recomputation
Yu Tang, Chenyu Wang, Yufan Zhang +5
The further development of deep neural networks is hampered by the limited GPU memory resource. Therefore, the optimization of GPU memory resources is highly demanded. Swapping and…