Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation

Abstract

Model-agnostic meta-reinforcement learning requires estimating the Hessian matrix of value functions. This is challenging from an implementation perspective, as repeatedly differentiating policy gradient estimates may lead to biased Hessian estimates. In this work, we provide a unifying framework for estimating higher-order derivatives of value functions, based on off-policy evaluation. Our framework interprets a number of prior approaches as special cases and elucidates the bias and variance trade-off of Hessian estimates. This framework also opens the door to a new family of estimates, which can be easily implemented with auto-differentiation libraries, and lead to performance gains in practice.

arXiv commentAccepted at Neural Information Processing Systems (NeurIPS), 2021. Code is available at https://github.com/robintyh1/neurips2021-meta-gradient-offpolicy-evaluation

References in corpus (19)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Dueling Network Architectures for Deep Reinforcement Learning
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Gradient Estimation Using Stochastic Computation Graphs
• Simple random search provides a competitive approach to reinforcement learning
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• Evolved Policy Gradients
• Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• Discovering Reinforcement Learning Algorithms
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• Meta-learning of Sequential Strategies
• A Self-Tuning Actor-Critic Algorithm
• Action-depedent Control Variates for Policy Optimization via Stein's Identity
• Adaptive Trade-Offs in Off-Policy Learning
• Taylor Expansion Policy Optimization