Program Synthesis Guided Reinforcement Learning for Partially Observed Environments

Abstract

A key challenge for reinforcement learning is solving long-horizon planning problems. Recent work has leveraged programs to guide reinforcement learning in these settings. However, these approaches impose a high manual burden on the user since they must provide a guiding program for every new task. Partially observed environments further complicate the programming task because the program must implement a strategy that correctly, and ideally optimally, handles every possible configuration of the hidden regions of the environment. We propose a new approach, model predictive program synthesis (MPPS), that uses program synthesis to automatically generate the guiding programs. It trains a generative model to predict the unobserved portions of the world, and then synthesizes a program based on samples from this model in a way that is robust to its uncertainty. In our experiments, we show that our approach significantly outperforms non-program-guided approaches on a set of challenging benchmarks, including a 2D Minecraft-inspired environment where the agent must complete a complex sequence of subtasks to achieve its goal, and achieves a similar performance as using handcrafted programs to guide the agent. Our results demonstrate that our approach can obtain the benefits of program-guided reinforcement learning without requiring the user to provide a new guiding program for every new task.

References in corpus (23)

• Auto-Encoding Variational Bayes
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• A Brief Survey of Deep Reinforcement Learning
• DeepCoder: Learning to Write Programs
• Modular Multitask Reinforcement Learning with Policy Sketches
• Verifiable Reinforcement Learning via Policy Extraction
• A Deep Hierarchical Approach to Lifelong Learning in Minecraft
• RobustFill: Neural Program Learning under Noisy I/O
• Hierarchical POMDP Controller Optimization by Likelihood Maximization
• Learning to Infer and Execute 3D Shape Programs
• Neural-Guided Deductive Search for Real-Time Program Synthesis from Examples
• Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis
• Towards Synthesizing Complex Programs from Input-Output Examples
• Hierarchical Reinforcement Learning for Zero-shot Generalization with Subtask Dependencies
• Program Synthesis with Large Language Models
• HOUDINI: Lifelong Learning as Program Synthesis
• A Composable Specification Language for Reinforcement Learning Tasks
• Write, Execute, Assess: Program Synthesis with a REPL
• Neural Guided Constraint Logic Programming for Program Synthesis
• Neurosymbolic Reinforcement Learning with Formally Verified Exploration
• Compositional Reinforcement Learning from Logical Specifications
• Neurosymbolic Transformers for Multi-Agent Communication
• Representing Partial Programs with Blended Abstract Semantics