Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning

Abstract

Self-supervised learning has shown great potentials in improving the video representation ability of deep neural networks by getting supervision from the data itself. However, some of the current methods tend to cheat from the background, i.e., the prediction is highly dependent on the video background instead of the motion, making the model vulnerable to background changes. To mitigate the model reliance towards the background, we propose to remove the background impact by adding the background. That is, given a video, we randomly select a static frame and add it to every other frames to construct a distracting video sample. Then we force the model to pull the feature of the distracting video and the feature of the original video closer, so that the model is explicitly restricted to resist the background influence, focusing more on the motion changes. We term our method as \emph{Background Erasing} (BE). It is worth noting that the implementation of our method is so simple and neat and can be added to most of the SOTA methods without much efforts. Specifically, BE brings 16.4% and 19.1% improvements with MoCo on the severely biased datasets UCF101 and HMDB51, and 14.5% improvement on the less biased dataset Diving48.

CVPR2021 camera ready

References in corpus (13)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• A Simple Framework for Contrastive Learning of Visual Representations
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• Contrastive Multiview Coding
• Self-supervised Co-training for Video Representation Learning
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework
• Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• Self-supervised Video Representation Learning by Pace Prediction
• Learning Discriminative Motion Features Through Detection

Cited by in corpus (2)

• DisCo: Remedy Self-supervised Learning on Lightweight Models with Distilled Contrastive Learning
• ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency