FanStore: Enabling Efficient and Scalable I/O for Distributed Deep Learning

Abstract

Emerging Deep Learning (DL) applications introduce heavy I/O workloads on computer clusters. The inherent long lasting, repeated, and random file access pattern can easily saturate the metadata and data service and negatively impact other users. In this paper, we present FanStore, a transient runtime file system that optimizes DL I/O on existing hardware/software stacks. FanStore distributes datasets to the local storage of compute nodes, and maintains a global namespace. With the techniques of system call interception, distributed metadata management, and generic data compression, FanStore provides a POSIX-compliant interface with native hardware throughput in an efficient and scalable manner. Users do not have to make intrusive code changes to use FanStore and take advantage of the optimized I/O. Our experiments with benchmarks and real applications show that FanStore can scale DL training to 512 compute nodes with over 90\% scaling efficiency.

References in corpus (4)

• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Horovod: fast and easy distributed deep learning in TensorFlow
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train