Regularized Off-Policy TD-Learning

Abstract

We present a novel $l_1$ regularized off-policy convergent TD-learning method (termed RO-TD), which is able to learn sparse representations of value functions with low computational complexity. The algorithmic framework underlying RO-TD integrates two key ideas: off-policy convergent gradient TD methods, such as TDC, and a convex-concave saddle-point formulation of non-smooth convex optimization, which enables first-order solvers and feature selection using online convex regularization. A detailed theoretical and experimental analysis of RO-TD is presented. A variety of experiments are presented to illustrate the off-policy convergence, sparse feature selection capability and low computational cost of the RO-TD algorithm.

26th Advances in Neural Information Processing Systems (NIPS). arXiv admin note: substantial text overlap with arXiv:1405.6757

References in corpus (4)

• Feature Selection Using Regularization in Approximate Linear Programs for Markov Decision Processes
• Greedy Algorithms for Sparse Reinforcement Learning
• A Dantzig Selector Approach to Temporal Difference Learning
• Sparse Q-learning with Mirror Descent

Cited by in corpus (10)

• Exploration-Enhanced POLITEX
• A generalization of regularized dual averaging and its dynamics
• Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient
• Implicit Temporal Differences
• Breaking the Deadly Triad with a Target Network
• Online Sparse Reinforcement Learning
• Trusted Neural Networks for Safety-Constrained Autonomous Control
• Causal Inference in Network Economics
• An Analysis of State-Relevance Weights and Sampling Distributions on L1-Regularized Approximate Linear Programming Approximation Accuracy
• Dantzig Selector with an Approximately Optimal Denoising Matrix and its Application to Reinforcement Learning