Experience Replay with Likelihood-free Importance Weights

Abstract

The use of past experiences to accelerate temporal difference (TD) learning of value functions, or experience replay, is a key component in deep reinforcement learning. Prioritization or reweighting of important experiences has shown to improve performance of TD learning algorithms.In this work, we propose to reweight experiences based on their likelihood under the stationary distribution of the current policy. Using the corresponding reweighted TD objective, we implicitly encourage small approximation errors on the value function over frequently encountered states. We use a likelihood-free density ratio estimator over the replay buffer to assign the prioritization weights. We apply the proposed approach empirically on two competitive methods, Soft Actor Critic (SAC) and Twin Delayed Deep Deterministic policy gradient (TD3) -- over a suite of OpenAI gym tasks and achieve superior sample complexity compared to other baseline approaches.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Prioritized Experience Replay
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Distributed Prioritized Experience Replay
• Sample Efficient Actor-Critic with Experience Replay
• Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)
• Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
• Safe and Efficient Off-Policy Reinforcement Learning
• Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past
• Planning by Prioritized Sweeping with Small Backups
• Neural Temporal-Difference and Q-Learning Provably Converge to Global Optima

Cited by in corpus (8)

• Online Coreset Selection for Rehearsal-based Continual Learning
• Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
• S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning
• Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective
• Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
• Lucid Dreaming for Experience Replay: Refreshing Past States with the Current Policy
• Model-Augmented Q-learning