Stage-based Hyper-parameter Optimization for Deep Learning

Abstract

As deep learning techniques advance more than ever, hyper-parameter optimization is the new major workload in deep learning clusters. Although hyper-parameter optimization is crucial in training deep learning models for high model performance, effectively executing such a computation-heavy workload still remains a challenge. We observe that numerous trials issued from existing hyper-parameter optimization algorithms share common hyper-parameter sequence prefixes, which implies that there are redundant computations from training the same hyper-parameter sequence multiple times. We propose a stage-based execution strategy for efficient execution of hyper-parameter optimization algorithms. Our strategy removes redundancy in the training process by splitting the hyper-parameter sequences of trials into homogeneous stages, and generating a tree of stages by merging the common prefixes. Our preliminary experiment results show that applying stage-based execution to hyper-parameter optimization algorithms outperforms the original trial-based method, saving required GPU-hours and end-to-end training time by up to 6.60 times and 4.13 times, respectively.

References in corpus (11)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• ADADELTA: An Adaptive Learning Rate Method
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Tune: A Research Platform for Distributed Model Selection and Training
• Improving Generalization Performance by Switching from Adam to SGD
• Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
• CHOPT : Automated Hyperparameter Optimization Framework for Cloud-Based Machine Learning Platforms
• MLtuner: System Support for Automatic Machine Learning Tuning