Imitation-Projected Programmatic Reinforcement Learning
arXiv:1907.05431
Abstract
We study the problem of programmatic reinforcement learning, in which policies are represented as short programs in a symbolic language. Programmatic policies can be more interpretable, generalizable, and amenable to formal verification than neural policies; however, designing rigorous learning approaches for such policies remains a challenge. Our approach to this challenge -- a meta-algorithm called PROPEL -- is based on three insights. First, we view our learning task as optimization in policy space, modulo the constraint that the desired policy has a programmatic representation, and solve this optimization problem using a form of mirror descent that takes a gradient step into the unconstrained policy space and then projects back onto the constrained space. Second, we view the unconstrained policy space as mixing neural and programmatic representations, which enables employing state-of-the-art deep policy gradient approaches. Third, we cast the projection step as program synthesis via imitation learning, and exploit contemporary combinatorial methods for this task. We present theoretical convergence results for PROPEL and empirically evaluate the approach in three continuous control domains. The experiments show that PROPEL can significantly outperform state-of-the-art approaches for learning programmatic policies.
arXiv commentPublished in Advances in Neural Information Processing Systems (NeurIPS) 2019
References in corpus (20)
- Continuous control with deep reinforcement learning
- Trust Region Policy Optimization
- Deep Reinforcement Learning that Matters
- Safe Model-based Reinforcement Learning with Stability Guarantees
- Safe Exploration in Continuous Action Spaces
- DeepCoder: Learning to Write Programs
- Verifiable Reinforcement Learning via Policy Extraction
- Reinforcement and Imitation Learning via Interactive No-Regret Learning
- RobustFill: Neural Program Learning under Noisy I/O
- Neuro-Symbolic Program Synthesis
- Learning to Search Better Than Your Teacher
- Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction
- Fast Policy Learning through Imitation and Reinforcement
- Neural Sketch Learning for Conditional Program Generation
- Control Regularization for Reduced Variance Reinforcement Learning
- HOUDINI: Lifelong Learning as Program Synthesis
- Sparse Q-learning with Mirror Descent
- Dual Policy Iteration
- Smooth Imitation Learning for Online Sequence Prediction
- Predictor-Corrector Policy Optimization