VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training

Abstract

Pre-training video transformers on extra large-scale datasets is generally required to achieve premier performance on relatively small datasets. In this paper, we show that video masked autoencoders (VideoMAE) are data-efficient learners for self-supervised video pre-training (SSVP). We are inspired by the recent ImageMAE and propose customized video tube masking with an extremely high ratio. This simple design makes video reconstruction a more challenging self-supervision task, thus encouraging extracting more effective video representations during this pre-training process. We obtain three important findings on SSVP: (1) An extremely high proportion of masking ratio (i.e., 90% to 95%) still yields favorable performance of VideoMAE. The temporally redundant video content enables a higher masking ratio than that of images. (2) VideoMAE achieves impressive results on very small datasets (i.e., around 3k-4k videos) without using any extra data. (3) VideoMAE shows that data quality is more important than data quantity for SSVP. Domain shift between pre-training and target datasets is an important issue. Notably, our VideoMAE with the vanilla ViT can achieve 87.4% on Kinetics-400, 75.4% on Something-Something V2, 91.3% on UCF101, and 62.6% on HMDB51, without using any extra data. Code is available at https://github.com/MCG-NJU/VideoMAE.

NeurIPS 2022 camera-ready version

Cited by in corpus (12)

• DINOv2: Learning Robust Visual Features without Supervision
• HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition
• Masked Autoencoder for Self-Supervised Pre-training on Lidar Point Clouds
• Self-Supervised Neuron Segmentation with Multi-Agent Reinforcement Learning
• VTG-GPT: Tuning-Free Zero-Shot Video Temporal Grounding with GPT
• MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition
• A vector quantized masked autoencoder for audiovisual speech emotion recognition
• ObjectMix: Data Augmentation by Copy-Pasting Objects in Videos for Action Recognition
• PosMLP-Video: Spatial and Temporal Relative Position Encoding for Efficient Video Recognition
• Zero-Shot Skeleton-Based Action Recognition With Prototype-Guided Feature Alignment
• An Evaluation of Large Pre-Trained Models for Gesture Recognition using Synthetic Videos
• On Masked Pre-training and the Marginal Likelihood