Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction

Abstract

Improving the sample efficiency of reinforcement learning algorithms requires effective exploration. Following the principle of $\text{optimism in the face of uncertainty}$ (OFU), we train a separate exploration policy to maximize the approximate upper confidence bound of the critics in an off-policy actor-critic framework. However, this introduces extra differences between the replay buffer and the target policy regarding their stationary state-action distributions. To mitigate the off-policy-ness, we adapt the recently introduced DICE framework to learn a distribution correction ratio for off-policy RL training. In particular, we correct the training distribution for both policies and critics. Empirically, we evaluate our proposed method in several challenging continuous control tasks and show superior performance compared to state-of-the-art methods. We also conduct extensive ablation studies to demonstrate the effectiveness and rationality of the proposed method.

arXiv commentDeep RL Workshop, NeurIPS 2022

References in corpus (28)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Noisy Networks for Exploration
• VIME: Variational Information Maximizing Exploration
• Parameter Space Noise for Exploration
• Large-Scale Study of Curiosity-Driven Learning
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Exploration by Random Network Distillation
• Episodic Curiosity through Reachability
• Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Deep Reinforcement Learning and the Deadly Triad
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Learning to Perform Physics Experiments via Deep Reinforcement Learning
• Curiosity-Driven Experience Prioritization via Density Estimation
• DORA The Explorer: Directed Outreaching Reinforcement Action-Selection
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• Reinforcement Learning via Fenchel-Rockafellar Duality
• Off-Policy Evaluation via the Regularized Lagrangian
• Experience Replay with Likelihood-free Importance Weights
• Variance-Based Rewards for Approximate Bayesian Reinforcement Learning
• Multi-task Batch Reinforcement Learning with Metric Learning