Coarse to Fine Multi-Resolution Temporal Convolutional Network
arXiv:2105.10859
Abstract
Temporal convolutional networks (TCNs) are a commonly used architecture for temporal video segmentation. TCNs however, tend to suffer from over-segmentation errors and require additional refinement modules to ensure smoothness and temporal coherency. In this work, we propose a novel temporal encoder-decoder to tackle the problem of sequence fragmentation. In particular, the decoder follows a coarse-to-fine structure with an implicit ensemble of multiple temporal resolutions. The ensembling produces smoother segmentations that are more accurate and better-calibrated, bypassing the need for additional refinement modules. In addition, we enhance our training with a multi-resolution feature-augmentation strategy to promote robustness to varying temporal resolutions. Finally, to support our architecture and encourage further sequence coherency, we propose an action loss that penalizes misclassifications at the video level. Experiments show that our stand-alone architecture, together with our novel feature-augmentation strategy and new loss, outperforms the state-of-the-art on three temporal video segmentation benchmarks.
References in corpus (10)
- Rethinking Atrous Convolution for Semantic Image Segmentation
- Two-Stream Convolutional Networks for Action Recognition in Videos
- Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
- Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
- Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
- A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation
- Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment
- VideoGraph: Recognizing Minutes-Long Human Activities in Videos
- Feature Re-Learning with Data Augmentation for Video Relevance Prediction
- PIC: Permutation Invariant Convolution for Recognizing Long-range Activities