Curious iLQR: Resolving Uncertainty in Model-based RL

Abstract

Curiosity as a means to explore during reinforcement learning problems has recently become very popular. However, very little progress has been made in utilizing curiosity for learning control. In this work, we propose a model-based reinforcement learning (MBRL) framework that combines Bayesian modeling of the system dynamics with curious iLQR, an iterative LQR approach that considers model uncertainty. During trajectory optimization the curious iLQR attempts to minimize both the task-dependent cost and the uncertainty in the dynamics model. We demonstrate the approach on reaching tasks with 7-DoF manipulators in simulation and on a real robot. Our experiments show that MBRL with curious iLQR reaches desired end-effector targets more reliably and with less system rollouts when learning a new task from scratch, and that the learned model generalizes better to new reaching tasks.

References in corpus (5)

• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Model-Based Active Exploration
• Curiosity Driven Exploration of Learned Disentangled Goal Spaces
• Intrinsic Motivation and Mental Replay enable Efficient Online Adaptation in Stochastic Recurrent Networks
• Risk Sensitive, Nonlinear Optimal Control: Iterative Linear Exponential-Quadratic Optimal Control with Gaussian Noise

Cited by in corpus (5)

• Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction
• Bayesian Curiosity for Efficient Exploration in Reinforcement Learning
• Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning
• Iterative Linear Quadratic Regulator With Variational Equation-Based Discretization
• Neural-iLQR: A Learning-Aided Shooting Method for Trajectory Optimization