Modular Deep Reinforcement Learning with Temporal Logic Specifications

Abstract

We propose an actor-critic, model-free, and online Reinforcement Learning (RL) framework for continuous-state continuous-action Markov Decision Processes (MDPs) when the reward is highly sparse but encompasses a high-level temporal structure. We represent this temporal structure by a finite-state machine and construct an on-the-fly synchronised product with the MDP and the finite machine. The temporal structure acts as a guide for the RL agent within the product, where a modular Deep Deterministic Policy Gradient (DDPG) architecture is proposed to generate a low-level control policy. We evaluate our framework in a Mars rover experiment and we present the success rate of the synthesised policy.

arXiv commentarXiv admin note: text overlap with arXiv:1902.00778

References in corpus (8)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Modular Multitask Reinforcement Learning with Policy Sketches
• Strategic Attentive Writer for Learning Macro-Actions
• Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraints
• FAUST${}^2$: Formal Abstractions of Uncountable-STate STochastic processes
• Quantitative Approximation of the Probability Distribution of a Markov Process by Formal Abstractions

Cited by in corpus (10)

• Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning
• LTL2Action: Generalizing LTL Instructions for Multi-Task RL
• Deep reinforcement learning under signal temporal logic constraints using Lagrangian relaxation
• Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning
• Compositional Reinforcement Learning from Logical Specifications
• Continuous Motion Planning with Temporal Logic Specifications using Deep Neural Networks
• Formal Policy Synthesis for Continuous-Space Systems via Reinforcement Learning
• The Logical Options Framework
• Goal Kernel Planning: Linearly-Solvable Non-Markovian Policies for Logical Tasks with Goal-Conditioned Options
• Lifelong Reinforcement Learning with Temporal Logic Formulas and Reward Machines