Learning to Synthesize Programs as Interpretable and Generalizable Policies
arXiv:2108.13643
Abstract
Recently, deep reinforcement learning (DRL) methods have achieved impressive performance on tasks in a variety of domains. However, neural network policies produced with DRL methods are not human-interpretable and often have difficulty generalizing to novel scenarios. To address these issues, prior works explore learning programmatic policies that are more interpretable and structured for generalization. Yet, these works either employ limited policy representations (e.g. decision trees, state machines, or predefined program templates) or require stronger supervision (e.g. input/output state pairs or expert demonstrations). We present a framework that instead learns to synthesize a program, which details the procedure to solve a task in a flexible and expressive manner, solely from reward signals. To alleviate the difficulty of learning to compose programs to induce the desired agent behavior from scratch, we propose to first learn a program embedding space that continuously parameterizes diverse behaviors in an unsupervised manner and then search over the learned program embedding space to yield a program that maximizes the return for a given task. Experimental results demonstrate that the proposed framework not only learns to reliably synthesize task-solving programs but also outperforms DRL and program synthesis baselines while producing interpretable and more generalizable policies. We also justify the necessity of the proposed two-stage learning scheme as well as analyze various methods for learning the program embedding.
NeurIPS 2021. 53 pages, 16 figures, 12 tables. Website at https://clvrai.github.io/leaps/
References in corpus (20)
- Evaluating Large Language Models Trained on Code
- DeepCoder: Learning to Write Programs
- Distral: Robust Multitask Reinforcement Learning
- Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
- RobustFill: Neural Program Learning under Noisy I/O
- Neuro-Symbolic Program Synthesis
- Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation
- Graph-based, Self-Supervised Program Repair from Diagnostic Feedback
- Differentiable Programs with Neural Libraries
- Program Synthesis with Large Language Models
- Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings
- Write, Execute, Assess: Program Synthesis with a REPL
- Strong Generalization and Efficiency in Neural Programs
- ProTo: Program-Guided Transformer for Program-Guided Tasks
- Toward Multimodal Model-Agnostic Meta-Learning
- Generalization to New Actions in Reinforcement Learning
- To Follow or not to Follow: Selective Imitation Learning from Observations
- SpreadsheetCoder: Formula Prediction from Semi-structured Context
- Representing Partial Programs with Blended Abstract Semantics
- Leveraging Language to Learn Program Abstractions and Search Heuristics