DINOv2: Learning Robust Visual Features without Supervision

Abstract

The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.

References in corpus (45)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Distilling the Knowledge in a Neural Network
• A Simple Framework for Contrastive Learning of Visual Representations
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• LLaMA: Open and Efficient Foundation Language Models
• Bootstrap your own latent: A new approach to self-supervised Learning
• The Kinetics Human Action Video Dataset
• On the Opportunities and Risks of Foundation Models
• PaLM: Scaling Language Modeling with Pathways
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Unsupervised Representation Learning by Predicting Image Rotations
• Fine-Grained Visual Classification of Aircraft
• LAION-5B: An open large-scale dataset for training next generation image-text models
• BEiT: BERT Pre-Training of Image Transformers
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Particular object retrieval with integral max-pooling of CNN activations
• Training Compute-Optimal Large Language Models
• Reproducible scaling laws for contrastive language-image learning
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
• Do ImageNet Classifiers Generalize to ImageNet?
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• Learning to Generate Reviews and Discovering Sentiment
• Billion-scale semi-supervised learning for image classification
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
• iBOT: Image BERT Pre-Training with Online Tokenizer
• Unsupervised Learning by Predicting Noise
• Vision Transformer Adapter for Dense Predictions
• Symbolic Discovery of Optimization Algorithms
• Polarized Self-Attention: Towards High-quality Pixel-wise Regression
• Self-supervised Pretraining of Visual Features in the Wild
• Unsupervised Semantic Segmentation by Distilling Feature Correspondences
• Masked Autoencoders that Listen
• Does Object Recognition Work for Everyone?
• Efficient Self-supervised Vision Transformers for Representation Learning
• BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation
• Spreading vectors for similarity search
• Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision
• Mugs: A Multi-Granular Self-Supervised Learning Framework
• The Met Dataset: Instance-level Recognition for Artworks
• Learning Super-Features for Image Retrieval
• Efficient Sequence Packing without Cross-contamination: Accelerating Large Language Models without Impacting Performance
• Weighted Ensemble Self-Supervised Learning
• A Simple Recipe for Competitive Low-compute Self supervised Vision Models

Cited by in corpus (85)

• Very high resolution canopy height maps from RGB imagery using self-supervised vision transformer and convolutional decoder trained on Aerial Lidar
• A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
• UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility
• Exploring scalable medical image encoders beyond text supervision
• AstroCLIP: A Cross-Modal Foundation Model for Galaxies
• Enhancing Skin Disease Classification Leveraging Transformer-based Deep Learning Architectures and Explainable AI
• Vision Transformers Need Registers
• Tracking and Mapping in Medical Computer Vision: A Review
• SpectralEarth: Training Hyperspectral Foundation Models at Scale
• A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application
• Challenges for Monocular 6D Object Pose Estimation in Robotics
• Machine learning of microstructure--property relationships in materials leveraging microstructure representation from foundational vision transformers
• Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Art
• Enhancing Network Initialization for Medical AI Models Using Large-Scale, Unlabeled Natural Images
• Analyzing mixed construction and demolition waste in material recovery facilities: evolution, challenges, and applications of computer vision and deep learning
• Evaluating Pre-trained Convolutional Neural Networks and Foundation Models as Feature Extractors for Content-based Medical Image Retrieval
• Self-supervised visual learning in the low-data regime: a comparative evaluation
• Active Predictive Coding: A Unified Neural Framework for Learning Hierarchical World Models for Perception and Planning
• Lost and Found: Overcoming Detector Failures in Online Multi-Object Tracking
• Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
• Few-Shot Panoptic Segmentation With Foundation Models
• Brittleness of metallic glasses dictated by their state at the fragile-to-strong transition temperature
• Second FRCSyn-onGoing: Winning Solutions and Post-Challenge Analysis to Improve Face Recognition with Synthetic Data
• YOLOv8 for Defect Inspection of Hexagonal Directed Self-Assembly Patterns: A Data-Centric Approach
• Distill-SODA: Distilling Self-Supervised Vision Transformer for Source-Free Open-Set Domain Adaptation in Computational Pathology
• Foundation Models and Transformers for Anomaly Detection: A Survey
• Visual place recognition for aerial imagery: A survey
• SurGen: 1020 H&E-stained Whole Slide Images With Survival and Genetic Markers
• Low-resource finetuning of foundation models beats state-of-the-art in histopathology
• VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation
• Large-vocabulary forensic pathological analyses via prototypical cross-modal contrastive learning
• Open Scene Graphs for Open-World Object-Goal Navigation
• Explicit Semantic-Base-Empowered Communications for 6G Mobile Networks
• StreetSurfaceVis: a dataset of crowdsourced street-level imagery annotated by road surface type and quality
• Augmentation-aware Self-supervised Learning with Conditioned Projector
• Efficient Image Pre-Training with Siamese Cropped Masked Autoencoders
• Re-identification from histopathology images
• Incomplete Multimodal Industrial Anomaly Detection via Cross-Modal Distillation
• Preemptively Pruning Clever-Hans Strategies in Deep Neural Networks
• Hierarchical end-to-end autonomous navigation through few-shot waypoint detection
• Hyperspectral Dataset and Deep Learning methods for Waste from Electric and Electronic Equipment Identification (WEEE)
• EBDM: Exemplar-guided Image Translation with Brownian-bridge Diffusion Models
• DatUS^2: Data-driven Unsupervised Semantic Segmentation with Pre-trained Self-supervised Vision Transformer
• Tissue Concepts: supervised foundation models in computational pathology
• Saturn Platform: Foundation Model Operations and Generative AI for Financial Services
• Generalist Models in Medical Image Segmentation: A Survey and Performance Comparison with Task-Specific Approaches
• Tackling Copyright Issues in AI Image Generation Through Originality Estimation and Genericization
• Pediatric brain tumor classification using digital histopathology and deep learning: evaluation of SOTA methods on a multi-center Swedish cohort
• Evaluating the Effectiveness of Attack-Agnostic Features for Morphing Attack Detection
• A Spitting Image: Modular Superpixel Tokenization in Vision Transformers
• Robustness Tokens: Towards Adversarial Robustness of Transformers
• ID-Booth: Identity-consistent Face Generation with Diffusion Models
• fruit-SALAD: A Style Aligned Artwork Dataset to reveal similarity perception in image embeddings
• AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Driving
• ConditionNET: Learning Preconditions and Effects for Execution Monitoring
• Nonverbal Immediacy Analysis in Education: A Multimodal Computational Model
• Open-Set 3D Semantic Instance Maps for Vision Language Navigation -- O3D-SIM
• SHREC 2025: Protein surface shape retrieval including electrostatic potential
• Are CLIP features all you need for Universal Synthetic Image Origin Attribution?
• On convex decision regions in deep network representations
• Approaching human 3D shape perception with neurally mappable models
• Leveraging Self-Supervised Vision Transformers for Segmentation-based Transfer Function Design
• Visual Reasoning in Object-Centric Deep Neural Networks: A Comparative Cognition Approach
• Monocular pose estimation of articulated open surgery tools -- in the wild
• Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field
• CLIP-RL: Surgical Scene Segmentation Using Contrastive Language-Vision Pretraining & Reinforcement Learning
• Benchmarking histopathology foundation models in a multi-center dataset for skin cancer subtyping
• Visual hallucination detection in large vision-language models via evidential conflict
• Fast Fishing: Approximating BAIT for Efficient and Scalable Deep Active Image Classification
• Universal and Transferable Attacks on Pathology Foundation Models
• Learning to Be a Transformer to Pinpoint Anomalies
• Fréchet Denoised Distance: Enhancing Plausibility Evaluation for Generated Designs with Denoising Autoencoder
• Precision at Scale: Domain-Specific Datasets On-Demand
• Shape Your Ground: Refining Road Surfaces Beyond Planar Representations
• ESCT3D: Efficient and Selectively Controllable Text-Driven 3D Content Generation with Gaussian Splatting
• Physically Interpretable Probabilistic Domain Characterization
• Improved Single Camera BEV Perception Using Multi-Camera Training
• Three Pillars improving Vision Foundation Model Distillation for Lidar
• Long-Term Visual Localization in Dynamic Benthic Environments: A Dataset, Footprint-Based Ground Truth, and Visual Place Recognition Benchmark
• Deep Learning for Accurate Vision-based Catch Composition in Tropical Tuna Purse Seiners
• Visual Lifelog Retrieval through Captioning-Enhanced Interpretation
• Weighted Ensemble Models Are Strong Continual Learners
• From 2D to 3D: AISG-SLA Visual Localization Challenge
• Representation Synthesis by Probabilistic Many-Valued Logic Operation in Self-Supervised Learning
• Advancing Video Self-Supervised Learning via Image Foundation Models