General non-linear Bellman equations

Abstract

We consider a general class of non-linear Bellman equations. These open up a design space of algorithms that have interesting properties, which has two potential advantages. First, we can perhaps better model natural phenomena. For instance, hyperbolic discounting has been proposed as a mathematical model that matches human and animal data well, and can therefore be used to explain preference orderings. We present a different mathematical model that matches the same data, but that makes very different predictions under other circumstances. Second, the larger design space can perhaps lead to algorithms that perform better, similar to how discount factors are often used in practice even when the true objective is undiscounted. We show that many of the resulting Bellman operators still converge to a fixed point, and therefore that the resulting algorithms are reasonable and inherit many beneficial properties of their linear counterparts.

References in corpus (7)

• Dueling Network Architectures for Deep Reinforcement Learning
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• Observe and Look Further: Achieving Consistent Performance on Atari
• Hyperbolic Discounting and Learning over Multiple Horizons
• Statistics and Samples in Distributional Reinforcement Learning
• Learning to Predict Independent of Span

Cited by in corpus (4)

• A reinforcement learning approach to rare trajectory sampling
• Muesli: Combining Improvements in Policy Optimization
• Return-based Scaling: Yet Another Normalisation Trick for Deep RL
• A Natural Actor-Critic Algorithm with Downside Risk Constraints