Lipschitz Continuity in Model-based Reinforcement Learning

Abstract

We examine the impact of learning Lipschitz continuous models in the context of model-based reinforcement learning. We provide a novel bound on multi-step prediction error of Lipschitz models where we quantify the error using the Wasserstein metric. We go on to prove an error bound for the value-function estimate arising from Lipschitz models and show that the estimated value function is itself Lipschitz. We conclude with empirical results that show the benefits of controlling the Lipschitz constant of neural-network models.

Accepted for the 35th International Conference on Machine Learning (ICML 2018)

References in corpus (4)

• On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning
• A unified view of entropy-regularized Markov decision processes
• Norm-Based Capacity Control in Neural Networks
• Learning Multimodal Transition Dynamics for Model-Based Reinforcement Learning

Cited by in corpus (10)

• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• DeepMDP: Learning Continuous Latent Space Models for Representation Learning
• Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future
• Zooming for Efficient Model-Free Reinforcement Learning in Metric Spaces
• Learning to Predict Without Looking Ahead: World Models Without Forward Prediction
• Equivalence Between Wasserstein and Value-Aware Loss for Model-based Reinforcement Learning
• Reinforcement Learning with Function-Valued Action Spaces for Partial Differential Equation Control
• Towards a Simple Approach to Multi-step Model-based Reinforcement Learning
• Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended version
• An investigation of model-free planning