Data-efficient Hindsight Off-policy Option Learning

Abstract

We introduce Hindsight Off-policy Options (HO2), a data-efficient option learning algorithm. Given any trajectory, HO2 infers likely option choices and backpropagates through the dynamic programming inference procedure to robustly train all policy components off-policy and end-to-end. The approach outperforms existing option learning methods on common benchmarks. To better understand the option framework and disentangle benefits from both temporal and action abstraction, we evaluate ablations with flat policies and mixture policies with comparable optimization. The results highlight the importance of both types of abstraction as well as off-policy training and trust-region constraints, particularly in challenging, simulated 3D robot manipulation tasks from raw pixel inputs. Finally, we intuitively adapt the inference step to investigate the effect of increased temporal abstraction on training with pre-trained options and from scratch.

Published at ICML2021

References in corpus (9)

• Distral: Robust Multitask Reinforcement Learning
• Learning and Transfer of Modulated Locomotor Controllers
• Near-Optimal Representation Learning for Hierarchical Reinforcement Learning
• Relative Entropy Regularized Policy Iteration
• Learning Abstract Options
• The Intentional Unintentional Agent: Learning to Solve Many Continuous Control Tasks Simultaneously
• The Termination Critic
• Sub-policy Adaptation for Hierarchical Reinforcement Learning
• DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations

Cited by in corpus (3)

• Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning
• Solving Compositional Reinforcement Learning Problems via Task Reduction
• A Simple Approach to Continual Learning by Transferring Skill Parameters