Value Iteration in Continuous Actions, States and Time
arXiv:2105.04682
Abstract
Classical value iteration approaches are not applicable to environments with continuous states and actions. For such environments, the states and actions are usually discretized, which leads to an exponential increase in computational complexity. In this paper, we propose continuous fitted value iteration (cFVI). This algorithm enables dynamic programming for continuous states and actions with a known dynamics model. Leveraging the continuous-time formulation, the optimal policy can be derived for non-linear control-affine dynamics. This closed-form solution enables the efficient extension of value iteration to continuous environments. We show in non-linear control experiments that the dynamic programming solution obtains the same quantitative performance as deep reinforcement learning methods in simulation but excels when transferred to the physical system. The policy obtained by cFVI is more robust to changes in the dynamics despite using only a deterministic model and without explicitly incorporating robustness in the optimization. Videos of the physical system are available at \url{https://sites.google.com/view/value-iteration}.
arXiv commentAccepted at International Conference on Machine Learning (ICML) 2021
References in corpus (17)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations
- Safe Model-based Reinforcement Learning with Stability Guarantees
- A linear theory for control of non-linear stochastic systems
- Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
- Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
- Neural Lyapunov Control
- MushroomRL: Simplifying Reinforcement Learning Research
- Learning a Contact-Adaptive Controller for Robust, Efficient Legged Locomotion
- Learning Deep Stochastic Optimal Control Policies using Forward-Backward SDEs
- Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controls
- Optimal Reinforcement Learning for Gaussian Systems
- Safe Optimal Control Using Stochastic Barrier Functions and Deep Forward-Backward SDEs
- Model-based Path Integral Stochastic Control: A Bayesian Nonparametric Approach
- HJB Optimal Feedback Control with Deep Differential Value Functions and Action Constraints