Motion2Vec: Semi-Supervised Representation Learning from Surgical Videos
arXiv:2006.00545
Abstract
Learning meaningful visual representations in an embedding space can facilitate generalization in downstream tasks such as action segmentation and imitation. In this paper, we learn a motion-centric representation of surgical video demonstrations by grouping them into action segments/sub-goals/options in a semi-supervised manner. We present Motion2Vec, an algorithm that learns a deep embedding feature space from video observations by minimizing a metric learning loss in a Siamese network: images from the same action segment are pulled together while pushed away from randomly sampled images of other segments, while respecting the temporal ordering of the images. The embeddings are iteratively segmented with a recurrent neural network for a given parametrization of the embedding space after pre-training the Siamese network. We only use a small set of labeled video segments to semantically align the embedding space and assign pseudo-labels to the remaining unlabeled data by inference on the learned model parameters. We demonstrate the use of this representation to imitate surgical suturing motions from publicly available videos of the JIGSAWS dataset. Results give 85.5 % segmentation accuracy on average suggesting performance improvement over several state-of-the-art baselines, while kinematic pose imitation gives 0.94 centimeter error in position per observation on the test set. Videos, code and data are available at https://sites.google.com/view/motion2vec
IEEE International Conference on Robotics and Automation (ICRA), 2020
References in corpus (12)
- Generative Adversarial Imitation Learning
- Unsupervised Visual Representation Learning by Context Prediction
- Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
- Unsupervised Learning of Visual Representations using Videos
- Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation
- A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation
- Grasp2Vec: Learning Object Representations from Self-Supervised Grasping
- Metric Learning with Adaptive Density Discrimination
- Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation
- Temporal Relational Reasoning in Videos
- Deep Visual Foresight for Planning Robot Motion
- Learning Actionable Representations from Visual Observations