Discounted Reinforcement Learning Is Not an Optimization Problem

Abstract

Discounted reinforcement learning is fundamentally incompatible with function approximation for control in continuing tasks. It is not an optimization problem in its usual formulation, so when using function approximation there is no optimal policy. We substantiate these claims, then go on to address some misconceptions about discounting and its connection to the average reward formulation. We encourage researchers to adopt rigorous optimization approaches, such as maximizing average reward, for reinforcement learning in continuing tasks.

Accepted for presentation at the Optimization Foundations of Reinforcement Learning Workshop at NeurIPS 2019

References in corpus (2)

• Hyperbolic Discounting and Learning over Multiple Horizons
• Issues concerning realizability of Blackwell optimal policies in reinforcement learning

Cited by in corpus (10)

• Data-driven control of micro-climate in buildings: an event-triggered reinforcement learning approach
• A reinforcement learning approach to rare trajectory sampling
• Temporal-Logic-Based Reward Shaping for Continuing Reinforcement Learning Tasks
• Learning and Planning in Average-Reward Markov Decision Processes
• Reward Function Design for Crowd Simulation via Reinforcement Learning
• Optimizing the Long-Term Average Reward for Continuing MDPs: A Technical Report
• Scalable Multi-agent Reinforcement Learning Algorithm for Wireless Networks
• Model-Free Design of Stochastic LQR Controller from Reinforcement Learning and Primal-Dual Optimization Perspective
• Lifetime policy reuse and the importance of task capacity
• Planning with Expectation Models for Control