A Composable Specification Language for Reinforcement Learning Tasks

Abstract

Reinforcement learning is a promising approach for learning control policies for robot tasks. However, specifying complex tasks (e.g., with multiple objectives and safety constraints) can be challenging, since the user must design a reward function that encodes the entire task. Furthermore, the user often needs to manually shape the reward to ensure convergence of the learning algorithm. We propose a language for specifying complex control tasks, along with an algorithm that compiles specifications in our language into a reward function and automatically performs reward shaping. We implement our approach in a tool called SPECTRL, and show that it outperforms several state-of-the-art baselines.

Cited by in corpus (10)

• Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning
• LTL2Action: Generalizing LTL Instructions for Multi-Task RL
• Reinforcement Learning with Stochastic Reward Machines
• Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning
• Compositional Reinforcement Learning from Logical Specifications
• Counterfactuals Modulo Temporal Logics
• The Logical Options Framework
• Program Synthesis Guided Reinforcement Learning for Partially Observed Environments
• Abstract Value Iteration for Hierarchical Reinforcement Learning
• Lifelong Reinforcement Learning with Temporal Logic Formulas and Reward Machines