Learning to Synthesize Programs as Interpretable and Generalizable Policies
arXiv:2108.13643
Abstract
Recently, deep reinforcement learning (DRL) methods have achieved impressive performance on tasks in a variety of domains. However, neural network policies produced with DRL methods are not human-interpretable and often have difficulty generalizing to novel scenarios. To address these issues, prior works explore learning programmatic policies that are more interpretable and structured for generalization. Yet, these works either employ limited policy representations (e.g. decision trees, state machines, or predefined program templates) or require stronger supervision (e.g. input/output state pairs or expert demonstrations). We present a framework that instead learns to synthesize a program, which details the procedure to solve a task in a flexible and expressive manner, solely from reward signals. To alleviate the difficulty of learning to compose programs to induce the desired agent behavior from scratch, we propose to first learn a program embedding space that continuously parameterizes diverse behaviors in an unsupervised manner and then search over the learned program embedding space to yield a program that maximizes the return for a given task. Experimental results demonstrate that the proposed framework not only learns to reliably synthesize task-solving programs but also outperforms DRL and program synthesis baselines while producing interpretable and more generalizable policies. We also justify the necessity of the proposed two-stage learning scheme as well as analyze various methods for learning the program embedding.
arXiv commentNeurIPS 2021. 53 pages, 16 figures, 12 tables. Website at https://clvrai.github.io/leaps/
References in corpus (35)
- Auto-Encoding Variational Bayes
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Prototypical Networks for Few-shot Learning
- Evaluating Large Language Models Trained on Code
- Estimating Uncertainty and Interpretability in Deep Learning for Coronavirus (COVID-19) Detection
- DeepCoder: Learning to Write Programs
- Neural Programmer-Interpreters
- Neural GPUs Learn Algorithms
- Modular Multitask Reinforcement Learning with Policy Sketches
- Distral: Robust Multitask Reinforcement Learning
- Sorting and Transforming Program Repair Ingredients via Deep Learning Code Similarities
- Verifiable Reinforcement Learning via Policy Extraction
- Estimating Training Data Influence by Tracing Gradient Descent
- Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
- RobustFill: Neural Program Learning under Noisy I/O
- Neuro-Symbolic Program Synthesis
- Making Neural Programming Architectures Generalize via Recursion
- Programming with a Differentiable Forth Interpreter
- Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation
- Dynamic Neural Program Embedding for Program Repair
- Graph-based, Self-Supervised Program Repair from Diagnostic Feedback
- Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis
- Differentiable Programs with Neural Libraries
- Program Synthesis with Large Language Models
- Neural Program Synthesis with Priority Queue Training
- Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings
- Write, Execute, Assess: Program Synthesis with a REPL
- Strong Generalization and Efficiency in Neural Programs
- ProTo: Program-Guided Transformer for Program-Guided Tasks
- Toward Multimodal Model-Agnostic Meta-Learning
- Generalization to New Actions in Reinforcement Learning
- To Follow or not to Follow: Selective Imitation Learning from Observations
- SpreadsheetCoder: Formula Prediction from Semi-structured Context
- Representing Partial Programs with Blended Abstract Semantics
- Leveraging Language to Learn Program Abstractions and Search Heuristics