Exploiting Transformer in Sparse Reward Reinforcement Learning for Interpretable Temporal Logic Motion Planning

Abstract

Automaton based approaches have enabled robots to perform various complex tasks. However, most existing automaton based algorithms highly rely on the manually customized representation of states for the considered task, limiting its applicability in deep reinforcement learning algorithms. To address this issue, by incorporating Transformer into reinforcement learning, we develop a Double-Transformer-guided Temporal Logic framework (T2TL) that exploits the structural feature of Transformer twice, i.e., first encoding the LTL instruction via the Transformer module for efficient understanding of task instructions during the training and then encoding the context variable via the Transformer again for improved task performance. Particularly, the LTL instruction is specified by co-safe LTL. As a semantics-preserving rewriting operation, LTL progression is exploited to decompose the complex task into learnable sub-goals, which not only converts non-Markovian reward decision processes to Markovian ones, but also improves the sampling efficiency by simultaneous learning of multiple sub-tasks. An environment-agnostic LTL pre-training scheme is further incorporated to facilitate the learning of the Transformer module resulting in an improved representation of LTL. The simulation results demonstrate the effectiveness of the T2TL framework.

IEEE Robotics and Automation Letters

References in corpus (6)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning
• Robot Cooking with Stir-fry: Bimanual Non-prehensile Manipulation of Semi-fluid Objects
• Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logic
• Overcoming Exploration: Deep Reinforcement Learning for Continuous Control in Cluttered Environments from Temporal Logic Specifications

Cited by in corpus (2)

• Generative AI for Self-Adaptive Systems: State of the Art and Research Roadmap
• Exploiting Hybrid Policy in Reinforcement Learning for Interpretable Temporal Logic Manipulation