LTL2Action: Generalizing LTL Instructions for Multi-Task RL

Abstract

We address the problem of teaching a deep reinforcement learning (RL) agent to follow instructions in multi-task environments. Instructions are expressed in a well-known formal language -- linear temporal logic (LTL) -- and can specify a diversity of complex, temporally extended behaviours, including conditionals and alternative realizations. Our proposed learning approach exploits the compositional syntax and the semantics of LTL, enabling our RL agent to learn task-conditioned policies that generalize to new instructions, not observed during training. To reduce the overhead of learning LTL semantics, we introduce an environment-agnostic LTL pretraining scheme which improves sample-efficiency in downstream environments. Experiments on discrete and continuous domains target combinatorial task sets of up to $\sim 10^{39}$ unique tasks and demonstrate the strength of our approach in learning to solve (unseen) tasks, given LTL instructions.

arXiv commentInternational Conference on Machine Learning (ICML) 2021

References in corpus (19)

• Modular Multitask Reinforcement Learning with Policy Sketches
• Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning
• Learning a SAT Solver from Single-Bit Supervision
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Grounded Language Learning in a Simulated 3D World
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Language as an Abstraction for Hierarchical Deep Reinforcement Learning
• Gated-Attention Architectures for Task-Oriented Language Grounding
• Learning to Understand Goal Specifications by Modelling Reward
• Environment-Independent Task Specifications via GLTL
• Grounded Language Learning Fast and Slow
• Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies
• Modular Deep Reinforcement Learning with Temporal Logic Specifications
• A Composable Specification Language for Reinforcement Learning Tasks
• Planning With Uncertain Specifications (PUnS)
• Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning
• Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas
• Learning Branching Heuristics for Propositional Model Counting
• Learning a natural-language to LTL executable semantic parser for grounded robotics

Cited by in corpus (2)

• Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning
• Compositional Reinforcement Learning from Logical Specifications