Differentiable Robust LQR Layers

Abstract

This paper proposes a differentiable robust LQR layer for reinforcement learning and imitation learning under model uncertainty and stochastic dynamics. The robust LQR layer can exploit the advantages of robust optimal control and model-free learning. It provides a new type of inductive bias for stochasticity and uncertainty modeling in control systems. In particular, we propose an efficient way to differentiate through a robust LQR optimization program by rewriting it as a convex program (i.e. semi-definite program) of the worst-case cost. Based on recent work on using convex optimization inside neural network layers, we develop a fully differentiable layer for optimizing this worst-case cost, i.e. we compute the derivative of a performance measure w.r.t the model's unknown parameters, model uncertainty and stochasticity parameters. We demonstrate the proposed method on imitation learning and approximate dynamic programming on stochastic and uncertain domains. The experiment results show that the proposed method can optimize robust policies under uncertain situations, and are able to achieve a significantly better performance than existing methods that do not model uncertainty directly.

arXiv comment10 pages

References in corpus (12)

• Playing Atari with Deep Reinforcement Learning
• Trust Region Policy Optimization
• Learning Latent Dynamics for Planning from Pixels
• Differentiable MPC for End-to-end Planning and Control
• The Predictron: End-To-End Learning and Planning
• Differentiable Convex Optimization Layers
• Constrained Policy Optimization
• Differentiable Dynamic Programming for Structured Prediction and Attention
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning
• Online Linear Quadratic Control
• Differentiable Trust Region Layers for Deep Reinforcement Learning