Variance-Reduced Off-Policy Memory-Efficient Policy Search

Abstract

Off-policy policy optimization is a challenging problem in reinforcement learning (RL). The algorithms designed for this problem often suffer from high variance in their estimators, which results in poor sample efficiency, and have issues with convergence. A few variance-reduced on-policy policy gradient algorithms have been recently proposed that use methods from stochastic optimization to reduce the variance of the gradient estimate in the REINFORCE algorithm. However, these algorithms are not designed for the off-policy setting and are memory-inefficient, since they need to collect and store a large ``reference'' batch of samples from time to time. To achieve variance-reduced off-policy-stable policy optimization, we propose an algorithm family that is memory-efficient, stochastically variance-reduced, and capable of learning from off-policy samples. Empirical studies validate the effectiveness of the proposed approaches.

References in corpus (12)

• Continuous control with deep reinforcement learning
• Addressing Function Approximation Error in Actor-Critic Methods
• Deep Q-learning from Demonstrations
• The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Stochastic Recursive Gradient Algorithm for Nonconvex Optimization
• The Mirage of Action-Dependent Baselines in Reinforcement Learning
• Consistent On-Line Off-Policy Evaluation
• Sample Efficient Policy Gradient Methods with Recursive Variance Reduction
• Policy Optimization via Importance Sampling
• Generalized Emphatic Temporal Difference Learning: Bias-Variance Analysis
• Stochastic Recursive Momentum for Policy Gradient Methods

Cited by in corpus (2)

• Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality
• On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correction