BFTrainer: Low-Cost Training of Neural Networks on Unfillable Supercomputer Nodes

Abstract

Supercomputer FCFS-based scheduling policies result in many transient idle nodes, a phenomenon that is only partially alleviated by backfill scheduling methods that promote small jobs to run before large jobs. Here we describe how to realize a novel use for these otherwise wasted resources, namely, deep neural network (DNN) training. This important workload is easily organized as many small fragments that can be configured dynamically to fit essentially any node*time hole in a supercomputer's schedule. We describe how the task of rescaling suitable DNN training tasks to fit dynamically changing holes can be formulated as a deterministic mixed integer linear programming (MILP)-based resource allocation algorithm, and show that this MILP problem can be solved efficiently at run time. We show further how this MILP problem can be adapted to optimize for administrator- or user-defined metrics. We validate our method with supercomputer scheduler logs and different DNN training scenarios, and demonstrate efficiencies of up to 93% compared with running the same training tasks on dedicated nodes. Our method thus enables substantial supercomputer resources to be allocated to DNN training with no impact on other applications.

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Efficient Neural Architecture Search via Parameter Sharing
• Don't Decay the Learning Rate, Increase the Batch Size
• Horovod: fast and easy distributed deep learning in TensorFlow
• Hyp-RL : Hyperparameter Optimization by Reinforcement Learning
• Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning
• Deep Reinforcement Agent for Scheduling in HPC