Safe Interactive Model-Based Learning

Abstract

Control applications present hard operational constraints. A violation of these can result in unsafe behavior. This paper introduces Safe Interactive Model Based Learning (SiMBL), a framework to refine an existing controller and a system model while operating on the real environment. SiMBL is composed of the following trainable components: a Lyapunov function, which determines a safe set; a safe control policy; and a Bayesian RNN forward model. A min-max control framework, based on alternate minimisation and backpropagation through the forward model, is used for the offline computation of the controller and the safe set. Safety is formally verified a-posteriori with a probabilistic method that utilizes the Noise Contrastive Priors (NPC) idea to build a Bayesian RNN forward model with an additive state uncertainty estimate which is large outside the training data distribution. Iterative refinement of the model and the safe set is achieved thanks to a novel loss that conditions the uncertainty estimates of the new model to be close to the current one. The learned safe set and model can also be used for safe exploration, i.e., to collect data within the safe invariant set, for which a simple one-step MPC is proposed. The single components are tested on the simulation of an inverted pendulum with limited torque and stability region, showing that iteratively adding more data can improve the model, the controller and the size of the safe region.

NeurIPS 2019 workshop on Safety and Robustness in Decision-Making

References in corpus (20)

• Adam: A Method for Stochastic Optimization
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Gaussian Processes for Data-Efficient Learning in Robotics and Control
• InfoVAE: Information Maximizing Variational Autoencoders
• Safe Model-based Reinforcement Learning with Stability Guarantees
• Model-Ensemble Trust-Region Policy Optimization
• Variational Lossy Autoencoder
• Lyapunov-based Safe Policy Optimization for Continuous Control
• Variational Gaussian Process State-Space Models
• Model-Based Active Exploration
• The Lyapunov Neural Network: Adaptive Stability Certification for Safe Learning of Dynamical Systems
• Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
• Noise Contrastive Priors for Functional Uncertainty
• Learning-based Model Predictive Control for Safe Exploration and Reinforcement Learning
• End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Safety Augmented Value Estimation from Demonstrations (SAVED): Safe Deep Model-Based RL for Sparse Cost Robotic Tasks
• Towards Safe Reinforcement Learning Using NMPC and Policy Gradients: Part II - Deterministic Case
• Sampling-based verification of Lyapunov's inequality for piecewise continuous nonlinear systems
• Probabilistic model predictive safety certification for learning-based control

Cited by in corpus (3)

• Learning for Safety-Critical Control with Control Barrier Functions
• Neural Lyapunov Model Predictive Control: Learning Safe Global Controllers from Sub-optimal Examples
• Stabilizing Neural Control Using Self-Learned Almost Lyapunov Critics