Regularity and stability of feedback relaxed controls
arXiv:2001.03148
Abstract
This paper proposes a relaxed control regularization with general exploration rewards to design robust feedback controls for multi-dimensional continuous-time stochastic exit time problems. We establish that the regularized control problem admits a Hölder continuous feedback control, and demonstrate that both the value function and the feedback control of the regularized control problem are Lipschitz stable with respect to parameter perturbations. Moreover, we show that a pre-computed feedback relaxed control has a robust performance in a perturbed system, and derive a first-order sensitivity equation for both the value function and optimal feedback relaxed control. These stability results provide a theoretical justification for recent reinforcement learning heuristics that including an exploration reward in the optimization objective leads to more robust decision making. We finally prove first-order monotone convergence of the value functions for relaxed control problems with vanishing exploration parameters, which subsequently enables us to construct the pure exploitation strategy of the original control problem based on the feedback relaxed controls.
Additional comments have been included, such that the importance of stable feedback controls for reinforcement learning. The manuscript will be published in SIAM Journal on Control and Optimization
References in corpus (6)
- A Theory of Regularized Markov Decision Processes
- All Adapted Topologies are Equal
- Dynamic Programming Principles for Mean-Field Controls with Learning
- A neural network based policy iteration algorithm with global -superlinear convergence for stochastic games on domains
- Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon
- Accurate Computation of the Log-Sum-Exp and Softmax Functions