Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO

Abstract

We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms: Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). Specifically, we investigate the consequences of "code-level optimizations:" algorithm augmentations found only in implementations or described as auxiliary details to the core algorithm. Seemingly of secondary importance, such optimizations turn out to have a major impact on agent behavior. Our results show that they (a) are responsible for most of PPO's gain in cumulative reward over TRPO, and (b) fundamentally change how RL methods function. These insights show the difficulty and importance of attributing performance gains in deep reinforcement learning. Code for reproducing our results is available at https://github.com/MadryLab/implementation-matters .

arXiv commentICLR 2020 version. arXiv admin note: text overlap with arXiv:1811.02553

References in corpus (7)

• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Simple random search provides a competitive approach to reinforcement learning
• The Mirage of Action-Dependent Baselines in Reinforcement Learning
• Natural Environment Benchmarks for Reinforcement Learning
• Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods

Cited by in corpus (23)

• Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?
• Robust Reinforcement Learning on State Observations with Learned Optimal Adversary
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations
• Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning
• Learning to Iteratively Solve Routing Problems with Dual-Aspect Collaborative Transformer
• Experience Replay with Likelihood-free Importance Weights
• On the model-based stochastic value gradient for continuous reinforcement learning
• Observation Space Matters: Benchmark and Optimization Algorithm
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• End-to-End Reinforcement Learning of Koopman Models for Economic Nonlinear Model Predictive Control
• Differentiable Trust Region Layers for Deep Reinforcement Learning
• Constrained Model-Free Reinforcement Learning for Process Optimization
• Reward Function Design for Crowd Simulation via Reinforcement Learning
• A Pragmatic Look at Deep Imitation Learning
• Towards Deeper Deep Reinforcement Learning with Spectral Normalization
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
• Quantum Chemistry Driven Molecular Inverse Design with Data-free Reinforcement Learning
• A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets
• Attention or memory? Neurointerpretable agents in space and time
• General Characterization of Agents by States they Visit
• Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learning
• Direct Random Search for Fine Tuning of Deep Reinforcement Learning Policies
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control