34 citations · 47 across the 4 of their papers we have counts for
4 papers
Coop: Memory is not a Commodity
Jianhao Zhang, Shihan Ma, Peihong Liu +1
Tensor rematerialization allows the training of deep neural networks (DNNs) under limited memory budgets by checkpointing the models and recomputing the evicted tensors as needed.…
Comprehensive Deadlock Prevention for GPU Collective Communication
Lichen Pan, Juncheng Liu, Yongquan Fu +4
Distributed deep neural network training necessitates efficient GPU collective communications, which are inherently susceptible to deadlocks. GPU collective deadlocks arise easily…
AutoDDL: Automatic Distributed Deep Learning with Near-Optimal Bandwidth Cost
Jinfan Chen, Shigang Li, Ran Gun +2
Recent advances in deep learning are driven by the growing scale of computation, data, and models. However, efficiently training large-scale models on distributed systems requires…
OneFlow: Redesign the Distributed Deep Learning Framework from Scratch
Jinhui Yuan, Xinqi Li, Cheng Cheng +9
Deep learning frameworks such as TensorFlow and PyTorch provide a productive interface for expressing and training a deep neural network (DNN) model on a single device or using dat…