LIFT: Reinforcement Learning in Computer Systems by Learning From Demonstrations

Abstract

Reinforcement learning approaches have long appealed to the data management community due to their ability to learn to control dynamic behavior from raw system performance. Recent successes in combining deep neural networks with reinforcement learning have sparked significant new interest in this domain. However, practical solutions remain elusive due to large training data requirements, algorithmic instability, and lack of standard tools. In this work, we introduce LIFT, an end-to-end software stack for applying deep reinforcement learning to data management tasks. While prior work has frequently explored applications in simulations, LIFT centers on utilizing human expertise to learn from demonstrations, thus lowering online training times. We further introduce TensorForce, a TensorFlow library for applied deep reinforcement learning exposing a unified declarative interface to common RL algorithms, thus providing a backend to LIFT. We demonstrate the utility of LIFT in two case studies in database compound indexing and resource management in stream processing. Results show LIFT controllers initialized from demonstrations can outperform human baselines and heuristics across latency metrics and space usage by up to 70%.

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Efficient Estimation of Word Representations in Vector Space
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• TensorFlow: A system for large-scale machine learning
• Playing Atari with Deep Reinforcement Learning
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Asynchronous Methods for Deep Reinforcement Learning
• Snorkel: Rapid Training Data Creation with Weak Supervision
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Deep Reinforcement Learning: An Overview
• Device Placement Optimization with Reinforcement Learning
• Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
• Reinforcement Learning from Imperfect Demonstrations
• A Benchmark Environment Motivated by Industrial Control Problems
• The Case for Automatic Database Administration using Deep Reinforcement Learning
• A Machine Learning Approach to Routing

Cited by in corpus (9)

• Neo: A Learned Query Optimizer
• Bao: Learning to Steer Query Optimizers
• Towards a Hands-Free Query Optimizer through Deep Learning
• Learned Garbage Collection
• Learning Index Selection with Structured Action Spaces
• Improving Post-Processing of Audio Event Detectors Using Reinforcement Learning
• DeepCrawl: Deep Reinforcement Learning for Turn-based Strategy Games
• RLgraph: Modular Computation Graphs for Deep Reinforcement Learning
• RLCache: Automated Cache Management Using Reinforcement Learning