Deep Residual Reinforcement Learning

Abstract

We revisit residual algorithms in both model-free and model-based reinforcement learning settings. We propose the bidirectional target network technique to stabilize residual algorithms, yielding a residual version of DDPG that significantly outperforms vanilla DDPG in the DeepMind Control Suite benchmark. Moreover, we find the residual algorithm an effective approach to the distribution mismatch problem in model-based planning. Compared with the existing TD($k$) method, our residual-based method makes weaker assumptions about the model and yields a greater performance boost.

AAMAS 2020

References in corpus (3)

• Model-Based Reinforcement Learning via Meta-Policy Optimization
• Should one compute the Temporal Difference fix point or minimize the Bellman Residual? The unified oblique projection view
• A Kernel Loss for Solving the Bellman Equation

Cited by in corpus (2)

• Correcting Momentum in Temporal Difference Learning
• Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Models