Phasic Policy Gradient

Abstract

We introduce Phasic Policy Gradient (PPG), a reinforcement learning framework which modifies traditional on-policy actor-critic methods by separating policy and value function training into distinct phases. In prior methods, one must choose between using a shared network or separate networks to represent the policy and value function. Using separate networks avoids interference between objectives, while using a shared network allows useful features to be shared. PPG is able to achieve the best of both worlds by splitting optimization into two phases, one that advances training and one that distills features. PPG also enables the value function to be more aggressively optimized with a higher level of sample reuse. Compared to PPO, we find that PPG significantly improves sample efficiency on the challenging Procgen Benchmark.

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Dota 2 with Large Scale Deep Reinforcement Learning
• Solving Rubik's Cube with a Robot Hand
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Sample Efficient Actor-Critic with Experience Replay
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• A Comparative Analysis of Expected and Distributional Reinforcement Learning

Cited by in corpus (17)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Review
• Reinforcement Learning Algorithms: An Overview and Classification
• Fusion of Model-free Reinforcement Learning with Microgrid Control: Review and Vision
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
• Muesli: Combining Improvements in Policy Optimization
• Scaling Scaling Laws with Board Games
• Cross-Trajectory Representation Learning for Zero-Shot Generalization in RL
• Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation
• Prioritized Level Replay
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• Reinforcement Learning for Industrial Control Network Cyber Security Orchestration
• Batch size-invariance for policy optimization
• Learning to Design and Construct Bridge without Blueprint
• Diagnosing and exploiting the computational demands of videos games for deep reinforcement learning
• Mix and Mask Actor-Critic Methods