Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic

Abstract

Model-free deep reinforcement learning (RL) methods have been successful in a wide variety of simulated domains. However, a major obstacle facing deep RL in the real world is their high sample complexity. Batch policy gradient methods offer stable learning, but at the cost of high variance, which often requires large batches. TD-style methods, such as off-policy actor-critic and Q-learning, are more sample-efficient but biased, and often require costly hyperparameter sweeps to stabilize. In this work, we aim to develop methods that combine the stability of policy gradients with the efficiency of off-policy RL. We present Q-Prop, a policy gradient method that uses a Taylor expansion of the off-policy critic as a control variate. Q-Prop is both sample efficient and stable, and effectively combines the benefits of on-policy and off-policy methods. We analyze the connection between Q-Prop and existing model-free algorithms, and use control variate theory to derive two variants of Q-Prop with conservative and aggressive adaptation. We show that conservative Q-Prop provides substantial gains in sample efficiency over trust region policy optimization (TRPO) with generalized advantage estimation (GAE), and improves stability over deep deterministic policy gradient (DDPG), the state-of-the-art on-policy and off-policy methods, on OpenAI Gym's MuJoCo continuous control environments.

Conference Paper at the International Conference on Learning Representations (ICLR) 2017

References in corpus (7)

• Trust Region Policy Optimization
• Continuous Deep Q-Learning with Model-based Acceleration
• Doubly Robust Policy Evaluation and Learning
• Input Convex Neural Networks
• Neural Variational Inference and Learning in Belief Networks
• The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
• Variational Bayesian Inference with Stochastic Search

Cited by in corpus (56)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Soft Actor-Critic Algorithms and Applications
• Learning Combinatorial Optimization Algorithms over Graphs
• How Generative Adversarial Networks and Their Variants Work: An Overview
• MOPO: Model-based Offline Policy Optimization
• Differentiable MPC for End-to-end Planning and Control
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Simple random search provides a competitive approach to reinforcement learning
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• Preparing for the Unknown: Learning a Universal Policy with Online System Identification
• VIREL: A Variational Inference Framework for Reinforcement Learning
• Recall Traces: Backtracking Models for Efficient Reinforcement Learning
• ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replay
• Benchmark Environments for Multitask Learning in Continuous Domains
• Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges
• Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
• Hindsight Trust Region Policy Optimization
• A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning
• EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL
• Bi-level Off-policy Reinforcement Learning for Volt/VAR Control Involving Continuous and Discrete Devices
• Beyond variance reduction: Understanding the true impact of baselines on policy optimization
• P3O: Policy-on Policy-off Policy Optimization
• Human-in-the-Loop Methods for Data-Driven and Reinforcement Learning Systems
• Soft Policy Gradient Method for Maximum Entropy Deep Reinforcement Learning
• DOB-Net: Actively Rejecting Unknown Excessive Time-Varying Disturbances
• An Off-policy Policy Gradient Theorem Using Emphatic Weightings
• On the Search for Feedback in Reinforcement Learning
• Learning to Navigate Cloth using Haptics
• Reward-estimation variance elimination in sequential decision processes
• Enhanced Experience Replay Generation for Efficient Reinforcement Learning
• ANS: Adaptive Network Scaling for Deep Rectifier Reinforcement Learning Models
• Backprop-Q: Generalized Backpropagation for Stochastic Computation Graphs
• Transferable Active Grasping and Real Embodied Dataset
• Multi-Path Policy Optimization
• Merging Deterministic Policy Gradient Estimations with Varied Bias-Variance Tradeoff for Effective Deep Reinforcement Learning
• Zeroth-Order Supervised Policy Improvement
• On Proximal Policy Optimization's Heavy-tailed Gradients
• Hindsight Value Function for Variance Reduction in Stochastic Dynamic Environment
• Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning
• A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme
• Deep Reinforcement Learning for Playing 2.5D Fighting Games
• Extended Radial Basis Function Controller for Reinforcement Learning
• Critic PI2: Master Continuous Planning via Policy Improvement with Path Integrals and Deep Actor-Critic Reinforcement Learning
• Reinforcement learning with world model
• Costate-focused models for reinforcement learning
• IMPACT: Importance Weighted Asynchronous Architectures with Clipped Target Networks
• Discrete Action On-Policy Learning with Action-Value Critic
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Ranking Policy Gradient
• Survivable Robotic Control through Guided Bayesian Policy Search with Deep Reinforcement Learning
• Off-Policy Self-Critical Training for Transformer in Visual Paragraph Generation
• Combining Off and On-Policy Training in Model-Based Reinforcement Learning
• A2: Extracting Cyclic Switchings from DOB-nets for Rejecting Excessive Disturbances
• FiDi-RL: Incorporating Deep Reinforcement Learning with Finite-Difference Policy Search for Efficient Learning of Continuous Control
• Learning Physical-Layer Communication with Quantized Feedback