Prospect-theoretic Q-learning

Abstract

We consider a prospect theoretic version of the classical Q-learning algorithm for discounted reward Markov decision processes, wherein the controller perceives a distorted and noisy future reward, modeled by a nonlinearity that accentuates gains and underrepresents losses relative to a reference point. We analyze the asymptotic behavior of the scheme by analyzing its limiting differential equation and using the theory of monotone dynamical systems to infer its asymptotic behavior. Specifically, we show convergence to equilibria, and establish some qualitative facts about the equilibria themselves.

arXiv commentPublished in Systems and Control Letters. 16 pages, 8 figures

References in corpus (7)

• Risk-sensitive Reinforcement Learning
• Risk-sensitive Markov control processes
• Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Control
• Fragility of the Commons under Prospect-Theoretic Risk Attitudes
• Risk-Sensitive Reinforcement Learning: a Martingale Approach to Reward Uncertainty
• On the Geometry of Nash and Correlated Equilibria with Cumulative Prospect Theoretic Preferences
• Black-Box Strategies and Equilibrium for Games with Cumulative Prospect Theoretic Players