TAN: Temporal Aggregation Network for Dense Multi-label Action Recognition

Abstract

We present Temporal Aggregation Network (TAN) which decomposes 3D convolutions into spatial and temporal aggregation blocks. By stacking spatial and temporal convolutions repeatedly, TAN forms a deep hierarchical representation for capturing spatio-temporal information in videos. Since we do not apply 3D convolutions in each layer but only apply temporal aggregation blocks once after each spatial downsampling layer in the network, we significantly reduce the model complexity. The use of dilated convolutions at different resolutions of the network helps in aggregating multi-scale spatio-temporal information efficiently. Experiments show that our model is well suited for dense multi-label action recognition, which is a challenging sub-topic of action recognition that requires predicting multiple action labels in each frame. We outperform state-of-the-art methods by 5% and 3% on the Charades and Multi-THUMOS dataset respectively.

WACV 2019

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
• Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs
• Temporal Action Detection with Structured Segment Networks
• R-C3D: Region Convolutional 3D Network for Temporal Activity Detection
• Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos
• StyleBank: An Explicit Representation for Neural Image Style Transfer
• Temporal Context Network for Activity Localization in Videos
• CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos
• S3D: Single Shot multi-Span Detector via Fully 3D Convolutional Networks
• Decouple Learning for Parameterized Image Operators
• Dynamic Temporal Pyramid Network: A Closer Look at Multi-Scale Modeling for Activity Detection
• Deep Motion Boundary Detection