Masked Autoencoders Are Scalable Vision Learners

Abstract

This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3x or more) and improve accuracy. Our scalable approach allows for learning high-capacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream tasks outperforms supervised pre-training and shows promising scaling behavior.

arXiv commentTech report. arXiv v2: add more transfer learning results; v3: add robustness evaluation

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Decoupled Weight Decay Regularization
• How transferable are features in deep neural networks?
• Neural Discrete Representation Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• BEiT: BERT Pre-Training of Image Transformers
• CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
• Large Batch Training of Convolutional Networks
• Self-supervised Pretraining of Visual Features in the Wild
• Benchmarking Detection Transfer Learning with Vision Transformers
• VOLO: Vision Outlooker for Visual Recognition
• An Empirical Study of Training Self-Supervised Vision Transformers

Cited by in corpus (13)

• Attention Mechanisms in Computer Vision: A Survey
• Root-aligned SMILES: A Tight Representation for Chemical Reaction Prediction
• Benchmarking Detection Transfer Learning with Vision Transformers
• AstroCLIP: A Cross-Modal Foundation Model for Galaxies
• Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
• PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers
• RegionCL: Can Simple Region Swapping Contribute to Contrastive Learning?
• Bridging Global Context Interactions for High-Fidelity Image Completion
• Multi-Grid Graph Neural Networks with Self-Attention for Computational Mechanics
• Natural Scene Text Editing Based on AI
• MC-SSL0.0: Towards Multi-Concept Self-Supervised Learning
• Neural Optimization Kernel: Towards Robust Deep Learning
• Calibrated Self-supervised Vision Transformers Improve Intracranial Arterial Calcification Segmentation from Clinical CT Head Scans