Lifelong Reinforcement Learning with Temporal Logic Formulas and Reward Machines

Abstract

Continuously learning new tasks using high-level ideas or knowledge is a key capability of humans. In this paper, we propose Lifelong reinforcement learning with Sequential linear temporal logic formulas and Reward Machines (LSRM), which enables an agent to leverage previously learned knowledge to fasten learning of logically specified tasks. For the sake of more flexible specification of tasks, we first introduce Sequential Linear Temporal Logic (SLTL), which is a supplement to the existing Linear Temporal Logic (LTL) formal language. We then utilize Reward Machines (RM) to exploit structural reward functions for tasks encoded with high-level events, and propose automatic extension of RM and efficient knowledge transfer over tasks for continuous learning in lifetime. Experimental results show that LSRM outperforms the methods that learn the target tasks from scratch by taking advantage of the task decomposition using SLTL and knowledge transfer over RM during the lifelong learning process.

References in corpus (10)

• Experience Replay for Continual Learning
• Progress & Compress: A scalable framework for continual learning
• Modular Multitask Reinforcement Learning with Policy Sketches
• Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning
• Modular Deep Reinforcement Learning with Temporal Logic Specifications
• A Composable Specification Language for Reinforcement Learning Tasks
• A Boolean Task Algebra for Reinforcement Learning
• Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning
• Automata-Guided Hierarchical Reinforcement Learning for Skill Composition
• Disentangled Planning and Control in Vision Based Robotics via Reward Machines

Cited by in corpus (1)

• Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning