ParamCrop: Parametric Cubic Cropping for Video Contrastive Learning

Abstract

The central idea of contrastive learning is to discriminate between different instances and force different views from the same instance to share the same representation. To avoid trivial solutions, augmentation plays an important role in generating different views, among which random cropping is shown to be effective for the model to learn a generalized and robust representation. Commonly used random crop operation keeps the distribution of the difference between two views unchanged along the training process. In this work, we show that adaptively controlling the disparity between two augmented views along the training process enhances the quality of the learned representation. Specifically, we present a parametric cubic cropping operation, ParamCrop, for video contrastive learning, which automatically crops a 3D cubic by differentiable 3D affine transformations. ParamCrop is trained simultaneously with the video backbone using an adversarial objective and learns an optimal cropping strategy from the data. The visualizations show that ParamCrop adaptively controls the center distance and the IoU between two augmented views, and the learned change in the disparity along the training process is beneficial to learning a strong representation. Extensive ablation studies demonstrate the effectiveness of the proposed ParamCrop on multiple contrastive learning frameworks and video backbones. Codes and models will be available.

15 pages

References in corpus (15)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• What Makes for Good Views for Contrastive Learning?
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Large Batch Training of Convolutional Networks
• Exploring Simple Siamese Representation Learning
• Self-supervised Co-training for Video Representation Learning
• Self-Supervised MultiModal Versatile Networks
• Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• Self-supervised Video Representation Learning by Pace Prediction
• Viewmaker Networks: Learning Views for Unsupervised Representation Learning
• Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion