Optimizing for the Future in Non-Stationary MDPs
arXiv:2005.08158
Abstract
Most reinforcement learning methods are based upon the key assumption that the transition dynamics and reward functions are fixed, that is, the underlying Markov decision process is stationary. However, in many real-world applications, this assumption is violated, and using existing algorithms may result in a performance lag. To proactively search for a good future policy, we present a policy gradient algorithm that maximizes a forecast of future performance. This forecast is obtained by fitting a curve to the counter-factual estimates of policy performance over time, without explicitly modeling the underlying non-stationarity. The resulting algorithm amounts to a non-uniform reweighting of past data, and we observe that minimizing performance over some of the data from past episodes can be beneficial when searching for a policy that maximizes future performance. We show that our algorithm, called Prognosticator, is more robust to non-stationarity than two online adaptation techniques, on three simulated problems motivated by real-world applications.
Thirty-seventh International Conference on Machine Learning (ICML 2020)
References in corpus (17)
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic Algorithms and Applications
- A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environments
- On Upper-Confidence Bound Policies for Non-Stationary Bandit Problems
- Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
- Learning in Games: Robustness of Fast Convergence
- A Sliding-Window Algorithm for Markov Decision Processes with Arbitrarily Changing Rewards and Transitions
- Rotting Bandits
- Deep Reinforcement Learning amidst Lifelong Non-Stationarity
- Using Options and Covariance Testing for Long Horizon Off-Policy Policy Evaluation
- Adaptive Online Planning for Continual Lifelong Learning
- Nearly Optimal Algorithms for Piecewise-Stationary Cascading Bandits
- TIDBD: Adapting Temporal-difference Step-sizes Through Stochastic Meta-descent
- Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended version
- Learning and Planning for Time-Varying MDPs Using Maximum Likelihood Estimation
- Rotting bandits are not harder than stochastic ones
- An Optimistic Acceleration of AMSGrad for Nonconvex Optimization