Off-Policy Deep Reinforcement Learning without Exploration

Abstract

Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning with data uncorrelated to the distribution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.

arXiv commentICML 2019

References in corpus (16)

• Auto-Encoding Variational Bayes
• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Deep Q-learning from Demonstrations
• A Deeper Look at Experience Replay
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Reinforcement Learning from Imperfect Demonstrations
• Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction
• Fast Policy Learning through Imitation and Reinforcement
• Learning from Scarce Experience
• Randomized Value Functions via Multiplicative Normalizing Flows
• Constrained Exploration and Recovery from Experience Shaping

Cited by in corpus (33)

• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• MOReL : Model-Based Offline Reinforcement Learning
• Machine learning for industrial sensing and control: A survey and practical perspective
• Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation
• Optimism in Reinforcement Learning with Generalized Linear Function Approximation
• Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systems
• Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment
• Value Penalized Q-Learning for Recommender Systems
• The problem with DDPG: understanding failures in deterministic environments with sparse rewards
• On the model-based stochastic value gradient for continuous reinforcement learning
• Estimation Error Correction in Deep Reinforcement Learning for Deterministic Actor-Critic Methods
• DRIFT: Deep Reinforcement Learning for Functional Software Testing
• DraftRec: Personalized Draft Recommendation for Winning in Multi-Player Online Battle Arena Games
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• Is Deep Reinforcement Learning Ready for Practical Applications in Healthcare? A Sensitivity Analysis of Duel-DDQN for Hemodynamic Management in Sepsis Patients
• ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
• Why People Skip Music? On Predicting Music Skips using Deep Reinforcement Learning
• Sequential Evaluation and Generation Framework for Combinatorial Recommender System
• Adaptive Online Planning for Continual Lifelong Learning
• ORL-AUDITOR: Dataset Auditing in Offline Deep Reinforcement Learning
• A Low Latency Adaptive Coding Spiking Framework for Deep Reinforcement Learning
• Towards Off-Policy Reinforcement Learning for Ranking Policies with Human Feedback
• Offline Reinforcement Learning with Pseudometric Learning
• Disentangling Dynamics and Returns: Value Function Decomposition with Future Prediction
• DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning
• PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning
• OER: Offline Experience Replay for Continual Offline Reinforcement Learning
• Safe Policy Improvement Approaches on Discrete Markov Decision Processes
• SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situations
• Off-Policy Policy Gradient Algorithms by Constraining the State Distribution Shift
• Reinforcement Learning for Vision-based Object Manipulation with Non-parametric Policy and Action Primitives
• Enhancing Reinforcement Learning Through Guided Search
• Learning robust driving policies without online exploration