Adaptive Lambda Least-Squares Temporal Difference Learning

Abstract

Temporal Difference learning or TD($\lambda$) is a fundamental algorithm in the field of reinforcement learning. However, setting TD's $\lambda$ parameter, which controls the timescale of TD updates, is generally left up to the practitioner. We formalize the $\lambda$ selection problem as a bias-variance trade-off where the solution is the value of $\lambda$ that leads to the smallest Mean Squared Value Error (MSVE). To solve this trade-off we suggest applying Leave-One-Trajectory-Out Cross-Validation (LOTO-CV) to search the space of $\lambda$ values. Unfortunately, this approach is too computationally expensive for most practical applications. For Least Squares TD (LSTD) we show that LOTO-CV can be implemented efficiently to automatically tune $\lambda$ and apply function optimization methods to efficiently search the space of $\lambda$ values. The resulting algorithm, ALLSTD, is parameter free and our experiments demonstrate that ALLSTD is significantly computationally faster than the naïve LOTO-CV implementation while achieving similar performance.

References in corpus (1)

• Efficient Reinforcement Learning Using Recursive Least-Squares Methods

Cited by in corpus (6)

• A Self-Tuning Actor-Critic Algorithm
• Multi-step Off-policy Learning Without Importance Sampling Ratios
• Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies
• Temporal Difference Learning with Neural Networks - Study of the Leakage Propagation Problem
• Adaptive Temporal-Difference Learning for Policy Evaluation with Per-State Uncertainty Estimates
• META-Learning Eligibility Traces for More Sample Efficient Temporal Difference Learning