Deep Reinforcement Learning with Linear Quadratic Regulator Regions
arXiv:2002.09820
Abstract
Practitioners often rely on compute-intensive domain randomization to ensure reinforcement learning policies trained in simulation can robustly transfer to the real world. Due to unmodeled nonlinearities in the real system, however, even such simulated policies can still fail to perform stably enough to acquire experience in real environments. In this paper we propose a novel method that guarantees a stable region of attraction for the output of a policy trained in simulation, even for highly nonlinear systems. Our core technique is to use "bias-shifted" neural networks for constructing the controller and training the network in the simulator. The modified neural networks not only capture the nonlinearities of the system but also provably preserve linearity in a certain region of the state space and thus can be tuned to resemble a linear quadratic regulator that is known to be stable for the real system. We have tested our new method by transferring simulated policies for a swing-up inverted pendulum to real systems and demonstrated its efficacy.
References in corpus (13)
- Addressing Function Approximation Error in Actor-Critic Methods
- Learning agile and dynamic motor skills for legged robots
- On the Number of Linear Regions of Deep Neural Networks
- Solving Rubik's Cube with a Robot Hand
- Deep Reinforcement Learning: An Overview
- Deep Drone Racing: From Simulation to Reality with Domain Randomization
- On the number of response regions of deep feed forward networks with piece-wise linear activations
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
- Batch-normalized Maxout Network in Network
- Assessing Transferability from Simulation to Reality for Reinforcement Learning
- On the Design of LQR Kernels for Efficient Controller Learning
- Model-free Reinforcement Learning with Robust Stability Guarantee
- Continuous Control with Contexts, Provably