Attention-Privileged Reinforcement Learning

Abstract

Image-based Reinforcement Learning is known to suffer from poor sample efficiency and generalisation to unseen visuals such as distractors (task-independent aspects of the observation space). Visual domain randomisation encourages transfer by training over visual factors of variation that may be encountered in the target domain. This increases learning complexity, can negatively impact learning rate and performance, and requires knowledge of potential variations during deployment. In this paper, we introduce Attention-Privileged Reinforcement Learning (APRiL) which uses a self-supervised attention mechanism to significantly alleviate these drawbacks: by focusing on task-relevant aspects of the observations, attention provides robustness to distractors as well as significantly increased learning efficiency. APRiL trains two attention-augmented actor-critic agents: one purely based on image observations, available across training and transfer domains; and one with access to privileged information (such as environment states) available only during training. Experience is shared between both agents and their attention mechanisms are aligned. The image-based policy can then be deployed without access to privileged information. We experimentally demonstrate accelerated and more robust learning on a diverse set of domains, leading to improved final performance for environments both within and outside the training distribution.

arXiv commentPublished at Conference on Robot Learning (CoRL) 2020

References in corpus (17)

• Continuous control with deep reinforcement learning
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Learning Transferable Features with Deep Adaptation Networks
• Solving Rubik's Cube with a Robot Hand
• Domain Separation Networks
• DeepMind Control Suite
• Parameter Space Noise for Exploration
• Learning Latent Dynamics for Planning from Pixels
• Hindsight Experience Replay
• CARLA: An Open Urban Driving Simulator
• Reinforcement Learning with Augmented Data
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• Learning a visuomotor controller for real world robotic grasping using simulated depth images
• Deep Attention Recurrent Q-Network
• Multi-focus Attention Network for Efficient Deep Reinforcement Learning
• Mutual Alignment Transfer Learning
• Robust Visual Domain Randomization for Reinforcement Learning

Cited by in corpus (3)

• Privileged Information Dropout in Reinforcement Learning
• Robust Asymmetric Learning in POMDPs
• Pretrained Encoders are All You Need