Temporal Convolutional Networks for Action Segmentation and Detection

Abstract

The ability to identify and temporally segment fine-grained human actions throughout a video is crucial for robotics, surveillance, education, and beyond. Typical approaches decouple this problem by first extracting local spatiotemporal features from video frames and then feeding them into a temporal classifier that captures high-level temporal patterns. We introduce a new class of temporal models, which we call Temporal Convolutional Networks (TCNs), that use a hierarchy of temporal convolutions to perform fine-grained action segmentation or detection. Our Encoder-Decoder TCN uses pooling and upsampling to efficiently capture long-range temporal patterns whereas our Dilated TCN uses dilated convolutions. We show that TCNs are capable of capturing action compositions, segment durations, and long-range dependencies, and are over a magnitude faster to train than competing LSTM-based Recurrent Neural Networks. We apply these models to three challenging fine-grained datasets and show large improvements over the state of the art.

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Multi-Scale Context Aggregation by Dilated Convolutions
• Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
• Learning Spatiotemporal Features with 3D Convolutional Networks
• Human Action Recognition using Factorized Spatio-Temporal Convolutional Networks
• Connectionist Temporal Modeling for Weakly Supervised Action Labeling

Cited by in corpus (8)

• Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment
• Rethinking the Faster R-CNN Architecture for Temporal Action Localization
• Temporal Human Action Segmentation via Dynamic Clustering
• Dilated Temporal Relational Adversarial Network for Generic Video Summarization
• Follow the Attention: Combining Partial Pose and Object Motion for Fine-Grained Action Detection
• Attentive Sequence to Sequence Translation for Localizing Clips of Interest by Natural Language Descriptions
• Temporally Folded Convolutional Neural Networks for Sequence Forecasting
• Train, Diagnose and Fix: Interpretable Approach for Fine-grained Action Recognition