Average-reward model-free reinforcement learning: a systematic review and literature mapping

Abstract

Reinforcement learning is important part of artificial intelligence. In this paper, we review model-free reinforcement learning that utilizes the average reward optimality criterion in the infinite horizon setting. Motivated by the solo survey by Mahadevan (1996a), we provide an updated review of work in this area and extend it to cover policy-iteration and function approximation methods (in addition to the value-iteration and tabular counterparts). We present a comprehensive literature mapping. We also identify and discuss opportunities for future work.

arXiv comment36 pages, refined prelim and politer sections

References in corpus (19)

• Equivalence Between Policy Gradients and Soft Q-Learning
• The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
• On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift
• A unified view of entropy-regularized Markov decision processes
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• On the Sample Complexity of the Linear Quadratic Regulator
• Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
• Off-Policy Actor-Critic
• Model-Free Linear Quadratic Control via Reduction to Expert Prediction
• Whittle index based Q-learning for restless bandits with average reward
• A Finite Time Analysis of Two Time-Scale Actor Critic Methods
• Primal-Dual $\pi$ Learning: Sample Complexity and Sublinear Run Time for Ergodic Markov Decision Problems
• Non-asymptotic Analysis of Biased Stochastic Approximation Scheme
• Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processes
• Contrasting Exploration in Parameter and Action Space: A Zeroth-Order Optimization Perspective
• Learning and Planning in Average-Reward Markov Decision Processes
• Average Reward Adjusted Discounted Reinforcement Learning: Near-Blackwell-Optimal Policies for Real-World Applications
• Investigating Recurrence and Eligibility Traces in Deep Q-Networks
• A Model-free Learning Algorithm for Infinite-horizon Average-reward MDPs with Near-optimal Regret

Cited by in corpus (3)

• Learning and Planning in Average-Reward Markov Decision Processes
• Average-Reward Reinforcement Learning with Trust Region Methods
• Surprise Maximization: A Dynamic Programming Approach