Data-efficient visuomotor policy training using reinforcement learning and generative models
arXiv:2007.13134
Abstract
We present a data-efficient framework for solving visuomotor sequential decision-making problems which exploits the combination of reinforcement learning (RL) and latent variable generative models. Our framework trains deep visuomotor policies by introducing an action latent variable such that the feed-forward policy search can be divided into three parts: (i) training a sub-policy that outputs a distribution over the action latent variable given a state of the system, (ii) unsupervised training of a generative model that outputs a sequence of motor actions conditioned on the latent action variable, and (iii) supervised training of the deep visuomotor policy in an end-to-end fashion. Our approach enables safe exploration and alleviates the data-inefficiency problem as it exploits prior knowledge about valid sequences of motor actions. Moreover, we provide a set of measures for evaluation of generative models such that we are able to predict the performance of the RL policy training prior to the actual training on a physical robot. We define two novel measures of disentanglement and local linearity for assessing the quality of latent representations, and complement them with existing measures for assessment of the learned distribution. We experimentally determine the characteristics of different generative models that have the most influence on performance of the final policy training on a robotic picking task.
References in corpus (20)
- Auto-Encoding Variational Bayes
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Trust Region Policy Optimization
- Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
- Disentangling by Factorising
- Recent Advances in Autoencoder-Based Representation Learning
- Towards a Definition of Disentangled Representations
- Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Study
- Assessing Generative Models via Precision and Recall
- Are Disentangled Representations Helpful for Abstract Visual Reasoning?
- One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
- Learning and Querying Fast Generative Models for Reinforcement Learning
- Deep Imitative Models for Flexible Inference, Planning, and Control
- A Distributional View on Multi-Objective Policy Optimization
- RoboNet: Large-Scale Multi-Robot Learning
- Multi-Agent Reinforcement Learning with Multi-Step Generative Models
- OOGAN: Disentangling GAN with One-Hot Sampling and Orthogonal Regularization
- Learning a generative model for robot control using visual feedback