AC-Teach: A Bayesian Actor-Critic Method for Policy Learning with an Ensemble of Suboptimal Teachers

Abstract

The exploration mechanism used by a Deep Reinforcement Learning (RL) agent plays a key role in determining its sample efficiency. Thus, improving over random exploration is crucial to solve long-horizon tasks with sparse rewards. We propose to leverage an ensemble of partial solutions as teachers that guide the agent's exploration with action suggestions throughout training. While the setup of learning with teachers has been previously studied, our proposed approach - Actor-Critic with Teacher Ensembles (AC-Teach) - is the first to work with an ensemble of suboptimal teachers that may solve only part of the problem or contradict other each other, forming a unified algorithmic solution that is compatible with a broad range of teacher ensembles. AC-Teach leverages a probabilistic representation of the expected outcome of the teachers' and student's actions to direct exploration, reduce dithering, and adapt to the dynamically changing quality of the learner. We evaluate a variant of AC-Teach that guides the learning of a Bayesian DDPG agent on three tasks - path following, robotic pick and place, and robotic cube sweeping using a hook - and show that it improves largely on sampling efficiency over a set of baselines, both for our target scenario of unconstrained suboptimal teachers and for easier setups with optimal or single teachers. Additional results and videos at https://sites.google.com/view/acteach/home.

Conference on Robot Learning (CoRL) 2019

References in corpus (18)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Deep Q-learning from Demonstrations
• Learning Dexterous In-Hand Manipulation
• Query-Efficient Imitation Learning for End-to-End Autonomous Driving
• Reinforcement and Imitation Learning via Interactive No-Regret Learning
• Reinforcement and Imitation Learning for Diverse Visuomotor Skills
• Reinforcement Learning from Imperfect Demonstrations
• Towards Characterizing Divergence in Deep Q-Learning
• Residual Policy Learning
• Theoretically-Grounded Policy Advice from Multiple Teachers in Reinforcement Learning Settings with Applications to Negative Transfer
• Bayesian Policy Gradients via Alpha Divergence Dropout Inference
• An Optimal Online Method of Selecting Source Policies for Reinforcement Learning
• Context-Aware Policy Reuse
• Interactive Reinforcement Learning with Dynamic Reuse of Prior Knowledge from Human/Agent's Demonstration

Cited by in corpus (5)

• Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
• Adversarial Imitation Learning with Trajectorial Augmentation and Correction
• Bootstrapping Motor Skill Learning with Motion Planning
• Dual-Arm Adversarial Robot Learning
• Survivable Robotic Control through Guided Bayesian Policy Search with Deep Reinforcement Learning