Merging Deterministic Policy Gradient Estimations with Varied Bias-Variance Tradeoff for Effective Deep Reinforcement Learning

Abstract

Deep reinforcement learning (DRL) on Markov decision processes (MDPs) with continuous action spaces is often approached by directly training parametric policies along the direction of estimated policy gradients (PGs). Previous research revealed that the performance of these PG algorithms depends heavily on the bias-variance tradeoffs involved in estimating and using PGs. A notable approach towards balancing this tradeoff is to merge both on-policy and off-policy gradient estimations. However existing PG merging methods can be sample inefficient and are not suitable to train deterministic policies directly. To address these issues, this paper introduces elite PGs and strengthens their variance reduction effect by adopting elitism and policy consolidation techniques to regularize policy training based on policy behavioral knowledge extracted from elite trajectories. Meanwhile, we propose a two-step method to merge elite PGs and conventional PGs as a new extension of the conventional interpolation merging method. At both the theoretical and experimental levels, we show that both two-step merging and interpolation merging can induce varied bias-variance tradeoffs during policy training. They enable us to effectively use elite PGs and mitigate their performance impact on trained policies. Our experiments also show that two-step merging can outperform interpolation merging and several state-of-the-art algorithms on six benchmark control tasks.

References in corpus (26)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Tutorial on Variational Autoencoders
• Distributed Prioritized Experience Replay
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Deep Exploration via Bootstrapped DQN
• Lookahead Optimizer: k steps forward, 1 step back
• No More Pesky Learning Rates
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Distral: Robust Multitask Reinforcement Learning
• Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
• Evolved Policy Gradients
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• A Theory of Regularized Markov Decision Processes
• Combining policy gradient and Q-learning
• The Mirage of Action-Dependent Baselines in Reinforcement Learning
• Understanding Short-Horizon Bias in Stochastic Meta-Optimization
• Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates
• Continual Reinforcement Learning with Complex Synapses
• Smoothed Action Value Functions for Learning Gaussian Policies
• Importance Resampling for Off-policy Prediction