most citedtorchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models

32 citations · 32 across the 1 of their papers we have counts for

collaborators

1 paper

cs.DC2020★ 32 cited

torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models

Chiheon Kim, Heungsub Lee, Myungryong Jeong +5

We design and implement a ready-to-use library in PyTorch for performing micro-batch pipeline parallelism with checkpointing proposed by GPipe (Huang et al., 2019). In particular,…