Offline Reinforcement Learning with Pseudometric Learning
arXiv:2103.01948
Abstract
Offline Reinforcement Learning methods seek to learn a policy from logged transitions of an environment, without any interaction. In the presence of function approximation, and under the assumption of limited coverage of the state-action space of the environment, it is necessary to enforce the policy to visit state-action pairs close to the support of logged transitions. In this work, we propose an iterative procedure to learn a pseudometric (closely related to bisimulation metrics) from logged transitions, and use it to define this notion of closeness. We show its convergence and extend it to the function approximation setting. We then use this pseudometric to define a new lookup based bonus in an actor-critic algorithm: PLOFF. This bonus encourages the actor to stay close, in terms of the defined pseudometric, to the support of logged transitions. Finally, we evaluate the method on hand manipulation and locomotion tasks.
arXiv commentICML 2021
References in corpus (24)
- Scikit-learn: Machine Learning in Python
- Auto-Encoding Variational Bayes
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Trust Region Policy Optimization
- Addressing Function Approximation Error in Actor-Critic Methods
- Prioritized Experience Replay
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- OpenAI Gym
- Conservative Q-Learning for Offline Reinforcement Learning
- Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
- Deep Reinforcement Learning that Matters
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- Off-Policy Deep Reinforcement Learning without Exploration
- Challenges of Real-World Reinforcement Learning
- Behavior Regularized Offline Reinforcement Learning
- MOPO: Model-based Offline Policy Optimization
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
- Safe Policy Improvement with Baseline Bootstrapping
- Metrics for Finite Markov Decision Processes
- Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
- Primal Wasserstein Imitation Learning
- Plannable Approximations to MDP Homomorphisms: Equivariance under Actions