Model-based Reinforcement Learning from Signal Temporal Logic Specifications

Abstract

Techniques based on Reinforcement Learning (RL) are increasingly being used to design control policies for robotic systems. RL fundamentally relies on state-based reward functions to encode desired behavior of the robot and bad reward functions are prone to exploitation by the learning agent, leading to behavior that is undesirable in the best case and critically dangerous in the worst. On the other hand, designing good reward functions for complex tasks is a challenging problem. In this paper, we propose expressing desired high-level robot behavior using a formal specification language known as Signal Temporal Logic (STL) as an alternative to reward/cost functions. We use STL specifications in conjunction with model-based learning to design model predictive controllers that try to optimize the satisfaction of the STL specification over a finite time horizon. The proposed algorithm is empirically evaluated on simulations of robotic system such as a pick-and-place robotic arm, and adaptive cruise control for autonomous vehicles.

Submitted to ICRA 2021

References in corpus (11)

• Continuous control with deep reinforcement learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Gaussian Processes for Data-Efficient Learning in Robotics and Control
• Safe Model-based Reinforcement Learning with Stability Guarantees
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• AI Safety Gridworlds
• Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning
• Barrier-Certified Adaptive Reinforcement Learning with Applications to Brushbot Navigation
• Deep Dynamics Models for Learning Dexterous Manipulation
• Logically-Constrained Reinforcement Learning
• Safe Learning of Quadrotor Dynamics Using Barrier Certificates

Cited by in corpus (4)

• Follow The Rules: Online Signal Temporal Logic Tree Search for Guided Imitation Learning in Stochastic Domains
• Policy Synthesis for Metric Interval Temporal Logic with Probabilistic Distributions
• Safe Model-based Control from Signal Temporal Logic Specifications Using Recurrent Neural Networks
• Deep Reinforcement Learning Based Networked Control with Network Delays for Signal Temporal Logic Specifications