Anderson Acceleration for Reinforcement Learning

Abstract

Anderson acceleration is an old and simple method for accelerating the computation of a fixed point. However, as far as we know and quite surprisingly, it has never been applied to dynamic programming or reinforcement learning. In this paper, we explain briefly what Anderson acceleration is and how it can be applied to value iteration, this being supported by preliminary experiments showing a significant speed up of convergence, that we critically discuss. We also discuss how this idea could be applied more generally to (deep) reinforcement learning.

arXiv commentEuropean Workshop on Reinforcement Learning (EWRL 2018)

References in corpus (1)

• Interpolatron: Interpolation or Extrapolation Schemes to Accelerate Optimization for Deep Neural Networks

Cited by in corpus (6)

• First-Order Methods for Wasserstein Distributionally Robust MDP
• A proof that Anderson acceleration improves the convergence rate in linearly converging fixed point methods (but not in those converging quadratically)
• Nonmonotone Globalization for Anderson Acceleration via Adaptive Regularization
• From Convex Optimization to MDPs: A Review of First-Order, Second-Order and Quasi-Newton Methods for MDPs
• On Anderson acceleration for partially observable Markov decision processes
• Anderson Acceleration for Seismic Inversion