Real-Time Reinforcement Learning

Abstract

Markov Decision Processes (MDPs), the mathematical framework underlying most algorithms in Reinforcement Learning (RL), are often used in a way that wrongfully assumes that the state of an agent's environment does not change during action selection. As RL systems based on MDPs begin to find application in real-world safety critical situations, this mismatch between the assumptions underlying classical MDPs and the reality of real-time computation may lead to undesirable outcomes. In this paper, we introduce a new framework, in which states and actions evolve simultaneously and show how it is related to the classical MDP formulation. We analyze existing algorithms under the new real-time formulation and show why they are suboptimal when used in real-time. We then use those insights to create a new algorithm Real-Time Actor-Critic (RTAC) that outperforms the existing state-of-the-art continuous control algorithm Soft Actor-Critic both in real-time and non-real-time settings. Code and videos can be found at https://github.com/rmst/rtrl.

Neural Information Processing Systems (2019)

Cited by in corpus (10)

• An empirical investigation of the challenges of real-world reinforcement learning
• Revisiting State Augmentation methods for Reinforcement Learning with Stochastic Delays
• Learning to Fly via Deep Model-Based Reinforcement Learning
• TriFinger: An Open-Source Robot for Learning Dexterity
• Delay-Aware Multi-Agent Reinforcement Learning for Cooperative and Competitive Environments
• Delay-Aware Model-Based Reinforcement Learning for Continuous Control
• Acting in Delayed Environments with Non-Stationary Markov Policies
• Reinforcement Learning with Random Delays
• What Robot do I Need? Fast Co-Adaptation of Morphology and Control using Graph Neural Networks
• META-Learning Eligibility Traces for More Sample Efficient Temporal Difference Learning