Hyperbolic Discounting and Learning over Multiple Horizons

Abstract

Reinforcement learning (RL) typically defines a discount factor as part of the Markov Decision Process. The discount factor values future rewards by an exponential scheme that leads to theoretical convergence guarantees of the Bellman equation. However, evidence from psychology, economics and neuroscience suggests that humans and animals instead have hyperbolic time-preferences. In this work we revisit the fundamentals of discounting in RL and bridge this disconnect by implementing an RL agent that acts via hyperbolic discounting. We demonstrate that a simple approach approximates hyperbolic discount functions while still using familiar temporal-difference learning techniques in RL. Additionally, and independent of hyperbolic discounting, we make a surprising discovery that simultaneously learning value functions over multiple time-horizons is an effective auxiliary task which often improves over a strong value-based RL agent, Rainbow.

References in corpus (6)

• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Exploration by Random Network Distillation
• A Distributional Perspective on Reinforcement Learning
• Dopamine: A Research Framework for Deep Reinforcement Learning
• How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies
• Many-Goals Reinforcement Learning

Cited by in corpus (22)

• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• A Self-Tuning Actor-Critic Algorithm
• Discounted Reinforcement Learning Is Not an Optimization Problem
• Scalable Transfer Learning with Expert Models
• Fast Efficient Hyperparameter Tuning for Policy Gradients
• Reinforcement Learning Under Moral Uncertainty
• General non-linear Bellman equations
• Which Mutual-Information Representation Learning Objectives are Sufficient for Control?
• A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms
• Reward Function Design for Crowd Simulation via Reinforcement Learning
• Learning Long-Term Reward Redistribution via Randomized Return Decomposition
• C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks
• Latent-state models for precision medicine
• Learn to Exceed: Stereo Inverse Reinforcement Learning with Concurrent Policy Optimization
• On The Effect of Auxiliary Tasks on Representation Dynamics
• Sample Efficient Ensemble Learning with Catalyst.RL
• Adapting to Reward Progressivity via Spectral Reinforcement Learning
• Markov Decision Processes with Time-Varying Geometric Discounting
• Learning State Representations from Random Deep Action-conditional Predictions
• A Hormetic Approach to the Value-Loading Problem: Preventing the Paperclip Apocalypse?
• Work in Progress: Temporally Extended Auxiliary Tasks
• Time-Aware Q-Networks: Resolving Temporal Irregularity for Deep Reinforcement Learning