Learning Compositional Neural Programs for Continuous Control

Abstract

We propose a novel solution to challenging sparse-reward, continuous control problems that require hierarchical planning at multiple levels of abstraction. Our solution, dubbed AlphaNPI-X, involves three separate stages of learning. First, we use off-policy reinforcement learning algorithms with experience replay to learn a set of atomic goal-conditioned policies, which can be easily repurposed for many tasks. Second, we learn self-models describing the effect of the atomic policies on the environment. Third, the self-models are harnessed to learn recursive compositional programs with multiple levels of abstraction. The key insight is that the self-models enable planning by imagination, obviating the need for interaction with the world when learning higher-level compositional programs. To accomplish the third stage of learning, we extend the AlphaNPI algorithm, which applies AlphaZero to learn recursive neural programmer-interpreters. We empirically show that AlphaNPI-X can effectively learn to tackle challenging sparse manipulation tasks, such as stacking multiple blocks, where powerful model-free baselines fail.

References in corpus (23)

• Continuous control with deep reinforcement learning
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Solving Rubik's Cube with a Robot Hand
• A Survey on Multi-Task Learning
• Learning Dexterous In-Hand Manipulation
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Data-Efficient Hierarchical Reinforcement Learning
• Neural Programmer-Interpreters
• Modular Multitask Reinforcement Learning with Policy Sketches
• Variational Intrinsic Control
• Automatic Goal Generation for Reinforcement Learning Agents
• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
• Efficient Exploration via State Marginal Matching
• Deep Dynamics Models for Learning Dexterous Manipulation
• Planning to Explore via Self-Supervised World Models
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Hierarchical Reinforcement Learning with Hindsight
• Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
• Learning Compositional Neural Programs with Recursive Tree Search and Planning
• Neural Task Programming: Learning to Generalize Across Hierarchical Tasks
• Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation
• Curiosity-Driven Multi-Criteria Hindsight Experience Replay
• Marginalized State Distribution Entropy Regularization in Policy Optimization