Continuous-Time Model-Based Reinforcement Learning

Abstract

Model-based reinforcement learning (MBRL) approaches rely on discrete-time state transition models whereas physical systems and the vast majority of control tasks operate in continuous-time. To avoid time-discretization approximation of the underlying process, we propose a continuous-time MBRL framework based on a novel actor-critic method. Our approach also infers the unknown state evolution differentials with Bayesian neural ordinary differential equations (ODE) to account for epistemic uncertainty. We implement and test our method on a new ODE-RL suite that explicitly solves continuous-time control systems. Our experiments illustrate that the model is robust against irregular and noisy data, is sample-efficient, and can solve control problems which pose challenges to discrete-time MBRL methods.

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• DeepMind Control Suite
• Hamiltonian Neural Networks
• Model-Ensemble Trust-Region Policy Optimization
• Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control
• Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control
• Scalable Gradients for Stochastic Differential Equations
• Functional Variational Bayesian Neural Networks
• Neural Jump Stochastic Differential Equations
• ODE${}^2$VAE: Deep generative second order ODEs with Bayesian neural networks
• BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning
• Policy Iterations for Reinforcement Learning Problems in Continuous Time and Space -- Fundamental Theory and Methods
• Learning unknown ODE models with Gaussian processes
• Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs
• Scalable Bayesian neural networks by layer-wise input augmentation