Learning Actionable Representations from Visual Observations

Abstract

In this work we explore a new approach for robots to teach themselves about the world simply by observing it. In particular we investigate the effectiveness of learning task-agnostic representations for continuous control tasks. We extend Time-Contrastive Networks (TCN) that learn from visual observations by embedding multiple frames jointly in the embedding space as opposed to a single frame. We show that by doing so, we are now able to encode both position and velocity attributes significantly more accurately. We test the usefulness of this self-supervised approach in a reinforcement learning setting. We show that the representations learned by agents observing themselves take random actions, or other agents perform tasks successfully, can enable the learning of continuous control policies using algorithms like Proximal Policy Optimization (PPO) using only the learned embeddings as input. We also demonstrate significant improvements on the real-world Pouring dataset with a relative error reduction of 39.4% for motion attributes and 11.1% for static attributes compared to the single-frame baseline. Video results are available at https://sites.google.com/view/actionablerepresentations .

arXiv commentThis work is accepted in IROS 2018. Project website: https://sites.google.com/view/actionablerepresentations

References in corpus (8)

• Continuous control with deep reinforcement learning
• DeepMind Control Suite
• Learning to Navigate in Complex Environments
• One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations
• Unsupervised state representation learning with robotic priors: a robustness benchmark
• Multi-task Learning for Continuous Control

Cited by in corpus (9)

• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Semantic Visual Navigation by Watching YouTube Videos
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• Motion Perception in Reinforcement Learning with Dynamic Objects
• Seeing All the Angles: Learning Multiview Manipulation Policies for Contact-Rich Tasks from Demonstrations
• Motion2Vec: Semi-Supervised Representation Learning from Surgical Videos
• Active Perception and Representation for Robotic Manipulation
• Leveraging Human Guidance for Deep Reinforcement Learning Tasks
• Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning