Device Placement Optimization with Reinforcement Learning

Abstract

The past few years have witnessed a growth in size and computational requirements for training and inference with neural networks. Currently, a common approach to address these requirements is to use a heterogeneous distributed environment with a mixture of hardware devices such as CPUs and GPUs. Importantly, the decision of placing parts of the neural models on devices is often made by human experts based on simple heuristics and intuitions. In this paper, we propose a method which learns to optimize device placement for TensorFlow computational graphs. Key to our method is the use of a sequence-to-sequence model to predict which subsets of operations in a TensorFlow graph should run on which of the available devices. The execution time of the predicted placements is then used as the reward signal to optimize the parameters of the sequence-to-sequence model. Our main result is that on Inception-V3 for ImageNet classification, and on RNN LSTM, for language modeling and neural machine translation, our model finds non-trivial device placements that outperform hand-crafted heuristics and traditional algorithmic methods.

To appear at ICML 2017

References in corpus (9)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• On the difficulty of training Recurrent Neural Networks
• Recurrent Neural Network Regularization
• Exploring the Limits of Language Modeling
• Deep Voice: Real-time Neural Text-to-Speech
• Listen, Attend and Spell
• ImageNet Large Scale Visual Recognition Challenge
• Achieving Budget-optimality with Adaptive Schemes in Crowdsourcing

Cited by in corpus (81)

• Deep Reinforcement Learning: An Overview
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
• A Survey of Machine Learning for Computer Architecture and Systems
• Learning to Optimize Tensor Programs
• Taskflow: A Lightweight Parallel and Heterogeneous Task Graph Computing System
• Database Meets Deep Learning: Challenges and Opportunities
• Supporting Very Large Models using Automatic Dataflow Graph Partitioning
• Beyond Data and Model Parallelism for Deep Neural Networks
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• Deep Learning in Mobile and Wireless Networking: A Survey
• Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Horizon: Facebook's Open Source Applied Reinforcement Learning Platform
• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• Dive into Deep Learning
• Task Placement and Resource Allocation for Edge Machine Learning: A GNN-based Multi-Agent Reinforcement Learning Paradigm
• Learning Scheduling Algorithms for Data Processing Clusters
• Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks
• Reinforcement Learning Applications
• TensorOpt: Exploring the Tradeoffs in Distributed DNN Training with Auto-Parallelism
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• Learning Collaborative Policies to Solve NP-hard Routing Problems
• Runtime Task Scheduling using Imitation Learning for Heterogeneous Many-Core Systems
• Learning the Travelling Salesperson Problem Requires Rethinking Generalization
• LIFT: Reinforcement Learning in Computer Systems by Learning From Demonstrations
• XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training
• Reinforced Genetic Algorithm Learning for Optimizing Computation Graphs
• GAP: Generalizable Approximate Graph Partitioning Framework
• torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models
• Constrained Deep Reinforcement Based Functional Split Optimization in Virtualized RANs
• AutoLoss: Learning Discrete Schedules for Alternate Optimization
• Communication-Efficient Edge AI: Algorithms and Systems
• Provably Efficient Q-Learning with Low Switching Cost
• GDP: Generalized Device Placement for Dataflow Graphs
• ProGraML: Graph-based Deep Learning for Program Optimization and Analysis
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• DL2: A Deep Learning-driven Scheduler for Deep Learning Clusters
• Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads
• Efficient Pipeline Planning for Expedited Distributed DNN Training
• Chameleon: Adaptive Code Optimization for Expedited Deep Neural Network Compilation
• Learning to Search via Retrospective Imitation
• Extract-Transform-Load for Video Streams
• Efficient Algorithms for Device Placement of DNN Graph Operators
• Electra: Conditional Generative Model based Predicate-Aware Query Approximation
• Constrained Combinatorial Optimization with Reinforcement Learning
• A General Large Neighborhood Search Framework for Solving Integer Linear Programs
• DeepSoCS: A Neural Scheduler for Heterogeneous System-on-Chip (SoC) Resource Scheduling
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Optimizing Task Placement and Online Scheduling for Distributed GNN Training Acceleration
• PaSE: Parallelization Strategies for Efficient DNN Training
• Polyjuice: High-Performance Transactions via Learned Concurrency Control
• ConfuciuX: Autonomous Hardware Resource Assignment for DNN Accelerators using Reinforcement Learning
• Packet Header Recognition Utilizing an All-Optical Reservoir Based on Reinforcement-Learning-Optimized Double-Ring Resonator
• Accelerated Training for CNN Distributed Deep Learning through Automatic Resource-Aware Layer Placement
• Compiler-Level Matrix Multiplication Optimization for Deep Learning
• Learning Index Selection with Structured Action Spaces
• Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training
• MARS: Malleable Actor-Critic Reinforcement Learning Scheduler
• Sentinel: Runtime Data Management on Heterogeneous Main MemorySystems for Deep Learning
• Automated Optical Multi-layer Design via Deep Reinforcement Learning
• Reinforcement Learning Enhanced Quantum-inspired Algorithm for Combinatorial Optimization
• RL-Scope: Cross-Stack Profiling for Deep Reinforcement Learning Workloads
• FTPipeHD: A Fault-Tolerant Pipeline-Parallel Distributed Training Framework for Heterogeneous Edge Devices
• Energy and Thermal-aware Resource Management of Cloud Data Centres: A Taxonomy and Future Directions
• HASCO: Towards Agile HArdware and Software CO-design for Tensor Computation
• Hydra: A Peer to Peer Distributed Training & Data Collection Framework
• Co-training for Policy Learning
• BoGraph: Structured Bayesian Optimization From Logs for Expensive Systems with Many Parameters
• Optimizing Memory Placement using Evolutionary Graph Reinforcement Learning
• Toward Efficient Online Scheduling for Distributed Machine Learning Systems
• A Deep Reinforcement Learning Approach to Multi-component Job Scheduling in Edge Computing
• DistIR: An Intermediate Representation and Simulator for Efficient Neural Network Distribution
• Scaling Distributed Deep Learning Workloads beyond the Memory Capacity with KARMA
• Self-Adaptive Reconfigurable Arrays (SARA): Using ML to Assist Scaling GEMM Acceleration
• Generating GPU Compiler Heuristics using Reinforcement Learning
• Intelligent Replication Management for HDFS Using Reinforcement Learning
• MQGrad: Reinforcement Learning of Gradient Quantization in Parameter Server
• Wield: Systematic Reinforcement Learning With Progressive Randomization
• Simulating Performance of ML Systems with Offline Profiling
• Transferable Graph Optimizers for ML Compilers
• Placement Optimization with Deep Reinforcement Learning