State Representation Learning from Demonstration

Abstract

Robots could learn their own state and world representation from perception and experience without supervision. This desirable goal is the main focus of our field of interest, state representation learning (SRL). Indeed, a compact representation of such a state is beneficial to help robots grasp onto their environment for interacting. The properties of this representation have a strong impact on the adaptive capability of the agent. In this article we present an approach based on imitation learning. The idea is to train several policies that share the same representation to reproduce various demonstrations. To do so, we use a multi-head neural network with a shared state representation feeding a task-specific agent. If the demonstrations are diverse, the trained representation will eventually contain the information necessary for all tasks, while discarding irrelevant information. As such, it will potentially become a compact state representation useful for new tasks. We call this approach SRLfD (State Representation Learning from Demonstration). Our experiments confirm that when a controller takes SRLfD-based representations as input, it can achieve better performance than with other representation strategies and promote more efficient reinforcement learning (RL) than with an end-to-end RL strategy.

arXiv commentPublished as a conference paper at LOD 2020

References in corpus (17)

• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Empirical Evaluation of Rectified Activations in Convolutional Network
• A guide to convolution arithmetic for deep learning
• DeepMind Control Suite
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Hindsight Experience Replay
• State Representation Learning for Control: An Overview
• One-Shot Visual Imitation Learning via Meta-Learning
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Time-Contrastive Networks: Self-Supervised Learning from Video
• PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations
• Data-Efficient Learning of Feedback Policies from Image Pixels using Deep Dynamical Models
• S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning
• Using PCA to Efficiently Represent State Spaces