Learning Temporally Invariant and Localizable Features via Data Augmentation for Video Recognition

Abstract

Deep-Learning-based video recognition has shown promising improvements along with the development of large-scale datasets and spatiotemporal network architectures. In image recognition, learning spatially invariant features is a key factor in improving recognition performance and robustness. Data augmentation based on visual inductive priors, such as cropping, flipping, rotating, or photometric jittering, is a representative approach to achieve these features. Recent state-of-the-art recognition solutions have relied on modern data augmentation strategies that exploit a mixture of augmentation operations. In this study, we extend these strategies to the temporal dimension for videos to learn temporally invariant or temporally localizable features to cover temporal perturbations or complex actions in videos. Based on our novel temporal data augmentation algorithms, video recognition performances are improved using only a limited amount of training data compared to the spatial-only data augmentation algorithms, including the 1st Visual Inductive Priors (VIPriors) for data-efficient action recognition challenge. Furthermore, learned features are temporally localizable that cannot be achieved using spatial augmentation algorithms. Our source code is available at https://github.com/taeoh-kim/temporal_data_augmentation.

European Conference on Computer Vision (ECCV) 2020, 1st Visual Inductive Priors for Data-Efficient Deep Learning Workshop (Oral)

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• A Simple Framework for Contrastive Learning of Visual Representations
• mixup: Beyond Empirical Risk Minimization
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Improved Regularization of Convolutional Neural Networks with Cutout
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Unsupervised Data Augmentation for Consistency Training
• Training Generative Adversarial Networks with Limited Data
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• DropBlock: A regularization method for convolutional networks
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Shake-Shake regularization
• ShakeDrop Regularization for Deep Residual Learning
• Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution
• Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition
• Differentiable Augmentation for Data-Efficient GAN Training
• AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures
• Improved Consistency Regularization for GANs
• Attribute Mix: Semantic Data Augmentation for Fine Grained Recognition
• Challenge report:VIPriors Action Recognition Challenge
• 2nd Place Scheme on Action Recognition Track of ECCV 2020 VIPriors Challenges: An Efficient Optical Flow Stream Guided Framework