VP-GO: a "light" action-conditioned visual prediction model

Abstract

Visual prediction models are a promising solution for visual-based robotic grasping of cluttered, unknown soft objects. Previous models from the literature are computationally greedy, which limits reproducibility; although some consider stochasticity in the prediction model, it is often too weak to catch the reality of robotics experiments involving grasping such objects. Furthermore, previous work focused on elementary movements that are not efficient to reason in terms of more complex semantic actions. To address these limitations, we propose VP-GO, a ``light'' stochastic action-conditioned visual prediction model. We propose a hierarchical decomposition of semantic grasping and manipulation actions into elementary end-effector movements, to ensure compatibility with existing models and datasets for visual prediction of robotic actions such as RoboNet. We also record and release a new open dataset for visual prediction of object grasping, called PandaGrasp. Our model can be pre-trained on RoboNet and fine-tuned on PandaGrasp, and performs similarly to more complex models in terms of signal prediction metrics. Qualitatively, it outperforms when predicting the outcome of complex grasps performed by our robot.

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Stochastic Video Generation with a Learned Prior
• Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics
• Unsupervised Learning for Physical Interaction through Video Prediction
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Stochastic Adversarial Video Prediction
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning
• Stochastic Variational Video Prediction
• Transformation-Based Models of Video Sequences
• Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours
• Video Imagination from a Single Image with Transformation Generation
• Video Frame Synthesis using Deep Voxel Flow
• Dense Optical Flow Prediction from a Static Image
• Real-Time Grasp Detection Using Convolutional Neural Networks