Characterizing Deep-Learning I/O Workloads in TensorFlow

Abstract

The performance of Deep-Learning (DL) computing frameworks rely on the performance of data ingestion and checkpointing. In fact, during the training, a considerable high number of relatively small files are first loaded and pre-processed on CPUs and then moved to accelerator for computation. In addition, checkpointing and restart operations are carried out to allow DL computing frameworks to restart quickly from a checkpoint. Because of this, I/O affects the performance of DL applications. In this work, we characterize the I/O performance and scaling of TensorFlow, an open-source programming framework developed by Google and specifically designed for solving DL problems. To measure TensorFlow I/O performance, we first design a micro-benchmark to measure TensorFlow reads, and then use a TensorFlow mini-application based on AlexNet to measure the performance cost of I/O and checkpointing in TensorFlow. To improve the checkpointing performance, we design and implement a burst buffer. We find that increasing the number of threads increases TensorFlow bandwidth by a maximum of 2.3x and 7.8x on our benchmark environments. The use of the tensorFlow prefetcher results in a complete overlap of computation on accelerator and input pipeline on CPU eliminating the effective cost of I/O on the overall performance. The use of a burst buffer to checkpoint to a fast small capacity storage and copy asynchronously the checkpoints to a slower large capacity storage resulted in a performance improvement of 2.6x with respect to checkpointing directly to slower storage on our benchmark environment.

Accepted for publication at pdsw-DISCS 2018

References in corpus (9)

• TensorFlow: A system for large-scale machine learning
• Caffe: Convolutional Architecture for Fast Feature Embedding
• NVIDIA Tensor Core Programmability, Performance & Precision
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• SparkNet: Training Deep Networks in Spark
• Benchmarking State-of-the-Art Deep Learning Software Tools
• The SAGE Project: a Storage Centric Approach for Exascale Computing
• Exploring Application Performance on Emerging Hybrid-Memory Supercomputers
• Exploring Scientific Application Performance Using Large Scale Object Storage

Cited by in corpus (11)

• Faster Neural Network Training with Data Echoing
• Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines
• A Selective Review on Statistical Methods for Massive Data Computation: Distributed Computing, Subsampling, and Minibatch Techniques
• tf-Darshan: Understanding Fine-grained I/O Performance in Machine Learning Workloads
• TensorFlow Doing HPC
• The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism
• Characterizing Deep Learning Training Workloads on Alibaba-PAI
• Stochastic Optimization with Laggard Data Pipelines
• An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models
• Extending Memory Capacity in Consumer Devices with Emerging Non-Volatile Memory: An Experimental Study
• Progressive Compressed Records: Taking a Byte out of Deep Learning Data