VideoDG: Generalizing Temporal Relations in Videos to Novel Domains

Abstract

This paper introduces video domain generalization where most video classification networks degenerate due to the lack of exposure to the target domains of divergent distributions. We observe that the global temporal features are less generalizable, due to the temporal domain shift that videos from other unseen domains may have an unexpected absence or misalignment of the temporal relations. This finding has motivated us to solve video domain generalization by effectively learning the local-relation features of different timescales that are more generalizable, and exploiting them along with the global-relation features to maintain the discriminability. This paper presents the VideoDG framework with two technical contributions. The first is a new deep architecture named the Adversarial Pyramid Network, which improves the generalizability of video features by capturing the local-relation, global-relation, and cross-relation features progressively. On the basis of pyramid features, the second contribution is a new and robust approach of adversarial data augmentation that can bridge different video domains by improving the diversity and quality of augmented data. We construct three video domain generalization benchmarks in which domains are divided according to different datasets, different consequences of actions, or different camera views, respectively. VideoDG consistently outperforms the combinations of previous video classification models and existing domain generalization methods on all benchmarks.

Accepted by IEEE TPAMI, 2021. Code: https://github.com/thuml/VideoDG

References in corpus (13)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• How transferable are features in deep neural networks?
• The Kinetics Human Action Video Dataset
• Theoretically Principled Trade-off between Robustness and Accuracy
• Spatiotemporal Residual Networks for Video Action Recognition
• Domain Generalization via Model-Agnostic Learning of Semantic Features
• Generalizing Across Domains via Cross-Gradient Training
• PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding
• Style Augmentation: Data Augmentation via Style Randomization
• In Search of Lost Domain Generalization
• Feature-Critic Networks for Heterogeneous Domain Generalization
• Temporal Attentive Alignment for Video Domain Adaptation