torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models

Chiheon Kim, Heungsub Lee, Myungryong Jeong +5

We design and implement a ready-to-use library in PyTorch for performing micro-batch pipeline parallelism with checkpointing proposed by GPipe (Huang et al., 2019). In particular,…