Multi-step Reinforcement Learning: A Unifying Algorithm

Abstract

Unifying seemingly disparate algorithmic ideas to produce better performing algorithms has been a longstanding goal in reinforcement learning. As a primary example, TD($\lambda$) elegantly unifies one-step TD prediction with Monte Carlo methods through the use of eligibility traces and the trace-decay parameter $\lambda$. Currently, there are a multitude of algorithms that can be used to perform TD control, including Sarsa, $Q$-learning, and Expected Sarsa. These methods are often studied in the one-step case, but they can be extended across multiple time steps to achieve better performance. Each of these algorithms is seemingly distinct, and no one dominates the others for all problems. In this paper, we study a new multi-step action-value algorithm called $Q(\sigma )$ which unifies and generalizes these existing algorithms, while subsuming them as special cases. A new parameter, $\sigma$, is introduced to allow the degree of sampling performed by the algorithm at each step during its backup to be continuously varied, with Sarsa existing at one extreme (full sampling), and Expected Sarsa existing at the other (pure expectation). $Q(\sigma )$ is generally applicable to both on- and off-policy learning, but in this work we focus on experiments in the on-policy case. Our results show that an intermediate value of $\sigma$, which results in a mixture of the existing algorithms, performs better than either extreme. The mixture can also be varied dynamically which can result in even greater performance.

Appeared at the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18)

Cited by in corpus (11)

• A reinforcement learning approach to rare trajectory sampling
• Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey
• The Effect of Multi-step Methods on Overestimation in Deep Reinforcement Learning
• Deep Reinforcement Learning for Optimal Stopping with Application in Financial Engineering
• Mixture of Step Returns in Bootstrapped DQN
• Unifying Ensemble Methods for Q-learning via Social Choice Theory
• Robust Temporal Difference Learning for Critical Domains
• Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatch
• Exploring TD error as a heuristic for $\sigma$ selection in Q($\sigma$, $\lambda$)
• Gradient Q$(\sigma ,\lambda )$: A Unified Algorithm with Function Approximation for Reinforcement Learning
• Context-aware Active Multi-Step Reinforcement Learning