A Study of Checkpointing in Large Scale Training of Deep Neural Networks

Abstract

Deep learning (DL) applications are increasingly being deployed on HPC systems, to leverage the massive parallelism and computing power of those systems for DL model training. While significant effort has been put to facilitate distributed training by DL frameworks, fault tolerance has been largely ignored. In this work, we evaluate checkpoint-restart, a common fault tolerance technique in HPC workloads. We perform experiments with three state-of-the-art DL frameworks common in HPC Chainer, PyTorch, and TensorFlow). We evaluate the computational cost of checkpointing, file formats and file sizes, the impact of scale, and deterministic checkpointing. Our evaluation shows some critical differences in checkpoint mechanisms and exposes several bottlenecks in existing checkpointing implementations. We provide discussion points that can aid users in selecting a fault-tolerant framework to use in HPC. We also provide takeaway points that framework developers can use to facilitate better checkpointing of DL workloads in HPC.

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning in Neural Networks: An Overview
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Wide Residual Networks
• One weird trick for parallelizing convolutional neural networks
• Horovod: fast and easy distributed deep learning in TensorFlow
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• PipeDream: Fast and Efficient Pipeline Parallel DNN Training
• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogates
• Deterministic Implementations for Reproducibility in Deep Reinforcement Learning
• Chainer: A Deep Learning Framework for Accelerating the Research Cycle
• Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory

Cited by in corpus (1)

• Let's Wait Awhile: How Temporal Workload Shifting Can Reduce Carbon Emissions in the Cloud