Reward Tweaking: Maximizing the Total Reward While Planning for Short Horizons

Abstract

In reinforcement learning, the discount factor $\gamma$ controls the agent's effective planning horizon. Traditionally, this parameter was considered part of the MDP; however, as deep reinforcement learning algorithms tend to become unstable when the effective planning horizon is long, recent works refer to $\gamma$ as a hyper-parameter -- thus changing the underlying MDP and potentially leading the agent towards sub-optimal behavior on the original task. In this work, we introduce \emph{reward tweaking}. Reward tweaking learns a surrogate reward function $\tilde{r}$ for the discounted setting that induces optimal behavior on the original finite-horizon total reward task. Theoretically, we show that there exists a surrogate reward that leads to optimality in the original task and discuss the robustness of our approach. Additionally, we perform experiments in high-dimensional continuous control tasks and show that reward tweaking guides the agent towards better long-horizon returns although it plans for short horizons.

arXiv commentPreprint

References in corpus (11)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Agent57: Outperforming the Atari Human Benchmark
• Meta-Gradient Reinforcement Learning
• Simple random search provides a competitive approach to reinforcement learning
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Sample Complexity of Episodic Fixed-Horizon Reinforcement Learning
• Beyond the One Step Greedy Approach in Reinforcement Learning
• Discount Factor as a Regularizer in Reinforcement Learning

Cited by in corpus (2)

• Heuristic-Guided Reinforcement Learning
• MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning