An empirical investigation of the challenges of real-world reinforcement learning

Abstract

Reinforcement learning (RL) has proven its worth in a series of artificial domains, and is beginning to show some successes in real-world scenarios. However, much of the research advances in RL are hard to leverage in real-world systems due to a series of assumptions that are rarely satisfied in practice. In this work, we identify and formalize a series of independent challenges that embody the difficulties that must be addressed for RL to be commonly deployed in real-world systems. For each challenge, we define it formally in the context of a Markov Decision Process, analyze the effects of the challenge on state-of-the-art learning algorithms, and present some existing attempts at tackling it. We believe that an approach that addresses our set of proposed challenges would be readily deployable in a large number of real world problems. Our proposed challenges are implemented in a suite of continuous control environments called the realworldrl-suite which we propose an as an open-source benchmark.

arXiv commentarXiv admin note: text overlap with arXiv:1904.12901

References in corpus (65)

• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic Algorithms and Applications
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Emergence of Locomotion Behaviours in Rich Environments
• DeepMind Control Suite
• Distributed Prioritized Experience Replay
• Deep Reinforcement Learning for Multi-objective Optimization
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Deep Reinforcement Learning that Matters
• Distributed Distributional Deterministic Policy Gradients
• Safe Exploration in Continuous Action Spaces
• Learning Dexterous In-Hand Manipulation
• Reward Constrained Policy Optimization
• Challenges of Real-World Reinforcement Learning
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Behavior Regularized Offline Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• Hybrid Reward Architecture for Reinforcement Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• MOReL : Model-Based Offline Reinforcement Learning
• A Deep Hierarchical Approach to Lifelong Learning in Minecraft
• Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning
• Constrained Policy Optimization
• A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation
• Safe Exploration in Finite Markov Decision Processes with Gaussian Processes
• RUDDER: Return Decomposition for Delayed Rewards
• Critic Regularized Regression
• Horizon: Facebook's Open Source Applied Reinforcement Learning Platform
• Making Contextual Decisions with Low Technical Debt
• Deep Dynamics Models for Learning Dexterous Manipulation
• Acme: A Research Framework for Distributed Reinforcement Learning
• Inverse Reward Design
• RecSim: A Configurable Simulation Platform for Recommender Systems
• Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• Responsive Safety in Reinforcement Learning by PID Lagrangian Methods
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• Adaptive Skills, Adaptive Partitions (ASAP)
• Policy Gradient for Coherent Risk Measures
• Unicorn: Continual Learning with a Universal, Off-policy Agent
• Behaviour Suite for Reinforcement Learning
• Deep Reinforcement Learning with a Natural Language Action Space
• BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning
• Value constrained model-free continuous control
• A Distributional View on Multi-Objective Policy Optimization
• Reactive Reinforcement Learning in Asynchronous Environments
• A Bayesian Approach to Robust Reinforcement Learning
• Model-Based Offline Planning
• Deep Robust Kalman Filter
• Real-Time Reinforcement Learning
• Learning Robust Options
• Iterative Hierarchical Optimization for Misspecified Problems (IHOMP)
• Action Assembly: Sparse Imitation Learning for Text Based Games with Combinatorial Action Spaces
• OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World
• On Ensuring that Intelligent Machines Are Well-Behaved
• Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Control
• Balancing Constraints and Rewards with Meta-Gradient D4PG
• Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification
• Situational Awareness by Risk-Conscious Skills

Cited by in corpus (17)

• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Pretraining Representations for Data-Efficient Reinforcement Learning
• A Self-Tuning Actor-Critic Algorithm
• CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
• Habitat 2.0: Training Home Assistants to Rearrange their Habitat
• Model-Based Offline Planning
• On the Emergence of Whole-body Strategies from Humanoid Robot Push-recovery Learning
• D2RL: Deep Dense Architectures in Reinforcement Learning
• Learning and Planning in Complex Action Spaces
• Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems
• Balancing Constraints and Rewards with Meta-Gradient D4PG
• Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification
• Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies
• Monte Carlo Tree Search for high precision manufacturing
• GrowSpace: Learning How to Shape Plants