PLAS: Latent Action Space for Offline Reinforcement Learning

Abstract

The goal of offline reinforcement learning is to learn a policy from a fixed dataset, without further interactions with the environment. This setting will be an increasingly more important paradigm for real-world applications of reinforcement learning such as robotics, in which data collection is slow and potentially dangerous. Existing off-policy algorithms have limited performance on static datasets due to extrapolation errors from out-of-distribution actions. This leads to the challenge of constraining the policy to select actions within the support of the dataset during training. We propose to simply learn the Policy in the Latent Action Space (PLAS) such that this requirement is naturally satisfied. We evaluate our method on continuous control benchmarks in simulation and a deformable object manipulation task with a physical robot. We demonstrate that our method provides competitive performance consistently across various continuous control tasks and different types of datasets, outperforming existing offline reinforcement learning methods with explicit constraints. Videos and code are available at https://sites.google.com/view/latent-policy.

References in corpus (9)

• Auto-Encoding Variational Bayes
• Continuous control with deep reinforcement learning
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Behavior Regularized Offline Reinforcement Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• MOReL : Model-Based Offline Reinforcement Learning
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Prediction and Control with Temporal Segment Models

Cited by in corpus (8)

• d3rlpy: An Offline Deep Reinforcement Learning Library
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
• Offline Reinforcement Learning with Reverse Model-based Imagination
• Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation
• Learning Off-Policy with Online Planning
• DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learning