TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentation

Abstract

Action segmentation as a milestone towards building automatic systems to understand untrimmed videos has received considerable attention in the recent years. It is typically being modeled as a sequence labeling problem but contains intrinsic and sufficient differences than text parsing or speech processing. In this paper, we introduce a novel hybrid temporal convolutional and recurrent network (TricorNet), which has an encoder-decoder architecture: the encoder consists of a hierarchy of temporal convolutional kernels that capture the local motion changes of different actions; the decoder is a hierarchy of recurrent neural networks that are able to learn and memorize long-term action dependencies after the encoding stage. Our model is simple but extremely effective in terms of video sequence labeling. The experimental results on three public action segmentation datasets have shown that the proposed model achieves superior performance over the state of the art.

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
• Learning Deconvolution Network for Semantic Segmentation
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos
• Connectionist Temporal Modeling for Weakly Supervised Action Labeling

Cited by in corpus (12)

• Gesture Recognition in Robotic Surgery: a Review
• Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment
• Action Segmentation with Joint Self-Supervised Temporal Domain Adaptation
• Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sources
• Weakly Supervised Energy-Based Learning for Action Segmentation
• Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery
• Coupled Generative Adversarial Network for Continuous Fine-grained Action Segmentation
• Action Segmentation with Mixed Temporal Domain Adaptation
• Fine-grained Action Segmentation using the Semi-Supervised Action GAN
• Symmetric Dilated Convolution for Surgical Gesture Recognition
• Hierarchical Attention Network for Action Segmentation
• Global2Local: Efficient Structure Search for Video Action Segmentation