Taylor Expansion Policy Optimization

Abstract

In this work, we investigate the application of Taylor expansions in reinforcement learning. In particular, we propose Taylor expansion policy optimization, a policy optimization formalism that generalizes prior work (e.g., TRPO) as a first-order special case. We also show that Taylor expansions intimately relate to off-policy evaluation. Finally, we show that this new formulation entails modifications which improve the performance of several state-of-the-art distributed algorithms.

References in corpus (12)

• Dueling Network Architectures for Deep Reinforcement Learning
• Dota 2 with Large Scale Deep Reinforcement Learning
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Distributed Prioritized Experience Replay
• Massively Parallel Methods for Deep Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• Observe and Look Further: Achieving Consistent Performance on Atari
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learning
• Policy Optimization via Importance Sampling
• Q($\lambda$) with Off-Policy Corrections
• Adaptive Trade-Offs in Off-Policy Learning

Cited by in corpus (2)

• An operator view of policy gradient methods
• Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation