Deep Predictive Policy Training using Reinforcement Learning

Abstract

Skilled robot task learning is best implemented by predictive action policies due to the inherent latency of sensorimotor processes. However, training such predictive policies is challenging as it involves finding a trajectory of motor activations for the full duration of the action. We propose a data-efficient deep predictive policy training (DPPT) framework with a deep neural network policy architecture which maps an image observation to a sequence of motor activations. The architecture consists of three sub-networks referred to as the perception, policy and behavior super-layers. The perception and behavior super-layers force an abstraction of visual and motor data trained with synthetic and simulated training samples, respectively. The policy super-layer is a small sub-network with fewer parameters that maps data in-between the abstracted manifolds. It is trained for each task using methods for policy search reinforcement learning. We demonstrate the suitability of the proposed architecture and learning framework by training predictive policies for skilled object grasping and ball throwing on a PR2 robot. The effectiveness of the method is illustrated by the fact that these tasks are trained using only about 180 real robot attempts with qualitative terminal rewards.

This work is submitted to IEEE/RSJ International Conference on Intelligent Robots and Systems 2017 (IROS2017)

References in corpus (8)

• Auto-Encoding Variational Bayes
• Continuous control with deep reinforcement learning
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Trust Region Policy Optimization
• Tutorial on Variational Autoencoders
• Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search
• Path Integral Guided Policy Search
• Self-learning and adaptation in a sensorimotor framework

Cited by in corpus (8)

• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
• Human-Robot Collaboration: From Psychology to Social Robotics
• Time Reversal as Self-Supervision
• Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight
• Deep Reinforcement Learning to Acquire Navigation Skills for Wheel-Legged Robots in Complex Environments
• ROS2Learn: a reinforcement learning framework for ROS 2
• DIPN: Deep Interaction Prediction Network with Application to Clutter Removal