V2CNet: A Deep Learning Framework to Translate Videos to Commands for Robotic Manipulation

Abstract

We propose V2CNet, a new deep learning framework to automatically translate the demonstration videos to commands that can be directly used in robotic applications. Our V2CNet has two branches and aims at understanding the demonstration video in a fine-grained manner. The first branch has the encoder-decoder architecture to encode the visual features and sequentially generate the output words as a command, while the second branch uses a Temporal Convolutional Network (TCN) to learn the fine-grained actions. By jointly training both branches, the network is able to model the sequential information of the command, while effectively encodes the fine-grained actions. The experimental results on our new large-scale dataset show that V2CNet outperforms recent state-of-the-art methods by a substantial margin, while its output can be applied in real robotic applications. The source code and trained models will be made available.

15 pages. arXiv admin note: substantial text overlap with arXiv:1710.00290

References in corpus (2)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

Cited by in corpus (3)

• Understanding Contexts Inside Robot and Human Manipulation Tasks through a Vision-Language Model and Ontology System in a Video Stream
• Understanding Action Sequences based on Video Captioning for Learning-from-Observation
• End-to-End Real-time Catheter Segmentation with Optical Flow-Guided Warping during Endovascular Intervention