Challenges of Real-World Reinforcement Learning

Abstract

Reinforcement learning (RL) has proven its worth in a series of artificial domains, and is beginning to show some successes in real-world scenarios. However, much of the research advances in RL are often hard to leverage in real-world systems due to a series of assumptions that are rarely satisfied in practice. We present a set of nine unique challenges that must be addressed to productionize RL to real world problems. For each of these challenges, we specify the exact meaning of the challenge, present some approaches from the literature, and specify some metrics for evaluating that challenge. An approach that addresses all nine challenges would be applicable to a large number of real world problems. We also present an example domain that has been modified to present these challenges as a testbed for practical RL research.

References in corpus (21)

• Continuous control with deep reinforcement learning
• Methods for Interpreting and Understanding Deep Neural Networks
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• DeepMind Control Suite
• Distributed Prioritized Experience Replay
• Deep Reinforcement Learning that Matters
• Learning Latent Dynamics for Planning from Pixels
• Doubly Robust Policy Evaluation and Learning
• Deep Q-learning from Demonstrations
• Safe Exploration in Continuous Action Spaces
• Reward Constrained Policy Optimization
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Hybrid Reward Architecture for Reinforcement Learning
• Policy Gradients with Variance Related Risk Criteria
• Programmatically Interpretable Reinforcement Learning
• Inverse Reward Design
• Policy Gradient for Coherent Risk Measures
• Value constrained model-free continuous control
• Deep Robust Kalman Filter
• Learning Robust Options
• Situational Awareness by Risk-Conscious Skills

Cited by in corpus (94)

• Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems
• Challenges in Deploying Machine Learning: a Survey of Case Studies
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environments
• Multi-UAV Path Planning for Wireless Data Harvesting with Deep Reinforcement Learning
• QPLEX: Duplex Dueling Multi-Agent Q-Learning
• A review on deep reinforcement learning for fluid mechanics: an update
• A review on Deep Reinforcement Learning for Fluid Mechanics
• An Optimistic Perspective on Offline Reinforcement Learning
• Critic Regularized Regression
• UAV Path Planning for Wireless Data Harvesting: A Deep Reinforcement Learning Approach
• ViNG: Learning Open-World Navigation with Visual Goals
• Language-guided Navigation via Cross-Modal Grounding and Alternate Adversarial Learning
• Deep reinforcement learning for guidewire navigation in coronary artery phantom
• RecSim: A Configurable Simulation Platform for Recommender Systems
• An empirical investigation of the challenges of real-world reinforcement learning
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Reinforcement Learning with Combinatorial Actions: An Application to Vehicle Routing
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• Gym-ANM: Reinforcement Learning Environments for Active Network Management Tasks in Electricity Distribution Systems
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• Safe Reinforcement Learning via Curriculum Induction
• Learning in situ: a randomized experiment in video streaming
• Learning hierarchical relationships for object-goal navigation
• Zero Time Waste: Recycling Predictions in Early Exit Neural Networks
• Gym-Ignition: Reproducible Robotic Simulations for Reinforcement Learning
• ABIDES-Gym: Gym Environments for Multi-Agent Discrete Event Simulation and Application to Financial Markets
• Representation Matters: Offline Pretraining for Sequential Decision Making
• IPO: Interior-point Policy Optimization under Constraints
• An Open-Source Multi-Goal Reinforcement Learning Environment for Robotic Manipulation with Pybullet
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning
• Risk-Averse Offline Reinforcement Learning
• D2RL: Deep Dense Architectures in Reinforcement Learning
• CAQL: Continuous Action Q-Learning
• Challenges of Applying Deep Reinforcement Learning in Dynamic Dispatching
• ACNMP: Skill Transfer and Task Extrapolation through Learning from Demonstration and Reinforcement Learning via Representation Sharing
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems
• GST: Group-Sparse Training for Accelerating Deep Reinforcement Learning
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research
• The Benchmark Lottery
• Probabilistic Active Meta-Learning
• Self-Imitation Advantage Learning
• Safe Reinforcement Learning with Natural Language Constraints
• I'm sorry Dave, I'm afraid I can't do that, Deep Q-learning from forbidden action
• A Framework for Studying Reinforcement Learning and Sim-to-Real in Robot Soccer
• Average-Reward Off-Policy Policy Evaluation with Function Approximation
• Continuous Doubly Constrained Batch Reinforcement Learning
• Balancing Constraints and Rewards with Meta-Gradient D4PG
• Constraint-Guided Reinforcement Learning: Augmenting the Agent-Environment-Interaction
• Offline Reinforcement Learning with Pseudometric Learning
• Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
• Breaking the Deadly Triad with a Target Network
• Model-free and Bayesian Ensembling Model-based Deep Reinforcement Learning for Particle Accelerator Control Demonstrated on the FERMI FEL
• PPMC RL Training Algorithm: Rough Terrain Intelligent Robots through Reinforcement Learning
• Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research
• Linear Representation Meta-Reinforcement Learning for Instant Adaptation
• Acting in Delayed Environments with Non-Stationary Markov Policies
• Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification
• Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study
• Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning
• Energy and Thermal-aware Resource Management of Cloud Data Centres: A Taxonomy and Future Directions
• Online Robust Control of Nonlinear Systems with Large Uncertainty
• Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch
• Truncated Emphatic Temporal Difference Methods for Prediction and Control
• Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learning
• Advancing Renewable Electricity Consumption With Reinforcement Learning
• Improving Learning from Demonstrations by Learning from Experience
• Aligning an optical interferometer with beam divergence control and continuous action space
• Boosting Offline Reinforcement Learning with Residual Generative Modeling
• World Programs for Model-Based Learning and Planning in Compositional State and Action Spaces
• Explaining Conditions for Reinforcement Learning Behaviors from Real and Imagined Data
• The Big Three: A Methodology to Increase Data Science ROI by Answering the Questions Companies Care About
• Quick Learner Automated Vehicle Adapting its Roadmanship to Varying Traffic Cultures with Meta Reinforcement Learning
• A survey of benchmarking frameworks for reinforcement learning
• Can Reinforcement Learning for Continuous Control Generalize Across Physics Engines?
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Reducing Conservativeness Oriented Offline Reinforcement Learning
• Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow
• Fault-Tolerant Federated Reinforcement Learning with Theoretical Guarantee
• How Do You Act? An Empirical Study to Understand Behavior of Deep Reinforcement Learning Agents
• Provable Multi-Objective Reinforcement Learning with Generative Models
• Learning robust driving policies without online exploration
• Model-free Policy Learning with Reward Gradients
• Dynamic Matching Markets in Power Grid: Concepts and Solution using Deep Reinforcement Learning
• Wield: Systematic Reinforcement Learning With Progressive Randomization
• Mapping the Internet: Modelling Entity Interactions in Complex Heterogeneous Networks
• Learning Robust Controllers Via Probabilistic Model-Based Policy Search
• Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning
• GrowSpace: Learning How to Shape Plants
• Robust Reinforcement Learning under model misspecification
• Resonance: Replacing Software Constants with Context-Aware Models in Real-time Communication
• Active Offline Policy Selection