Online Meta-Critic Learning for Off-Policy Actor-Critic Methods

Abstract

Off-Policy Actor-Critic (Off-PAC) methods have proven successful in a variety of continuous control tasks. Normally, the critic's action-value function is updated using temporal-difference, and the critic in turn provides a loss for the actor that trains it to take actions with higher expected return. In this paper, we introduce a novel and flexible meta-critic that observes the learning process and meta-learns an additional loss for the actor that accelerates and improves actor-critic learning. Compared to the vanilla critic, the meta-critic network is explicitly trained to accelerate the learning process; and compared to existing meta-learning algorithms, meta-critic is rapidly learned online for a single task, rather than slowly over a family of tasks. Crucially, our meta-critic framework is designed for off-policy based learners, which currently provide state-of-the-art reinforcement learning sample efficiency. We demonstrate that online meta-critic learning leads to improvements in avariety of continuous control environments when combined with contemporary Off-PAC methods DDPG, TD3 and the state-of-the-art SAC.

arXiv commentNeurIPS 2020

References in corpus (18)

• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Meta-Learning with Implicit Gradients
• Meta-Gradient Reinforcement Learning
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Evolved Policy Gradients
• Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
• Simulated Car Racing Championship: Competition Software Manual
• Learning to Teach with Dynamic Loss Functions
• Feature-Critic Networks for Heterogeneous Domain Generalization
• Discovery of Useful Questions as Auxiliary Tasks
• Learning Surrogate Losses
• Addressing the Loss-Metric Mismatch with Adaptive Loss Alignment

Cited by in corpus (2)

• Improved Robustness and Safety for Pre-Adaptation of Meta Reinforcement Learning with Prior Regularization
• Effective Regularization Through Loss-Function Metalearning