Unsupervised Representation Learning by Predicting Image Rotations

Abstract

Over the last years, deep convolutional neural networks (ConvNets) have transformed the field of computer vision thanks to their unparalleled capacity to learn high level semantic image features. However, in order to successfully learn those features, they usually require massive amounts of manually labeled data, which is both expensive and impractical to scale. Therefore, unsupervised semantic feature learning, i.e., learning without requiring manual annotation effort, is of crucial importance in order to successfully harvest the vast amount of visual data that are available today. In our work we propose to learn image features by training ConvNets to recognize the 2d rotation that is applied to the image that it gets as input. We demonstrate both qualitatively and quantitatively that this apparently simple task actually provides a very powerful supervisory signal for semantic feature learning. We exhaustively evaluate our method in various unsupervised feature learning benchmarks and we exhibit in all of them state-of-the-art performance. Specifically, our results on those benchmarks demonstrate dramatic improvements w.r.t. prior state-of-the-art approaches in unsupervised representation learning and thus significantly close the gap with supervised feature learning. For instance, in PASCAL VOC 2007 detection task our unsupervised pre-trained AlexNet model achieves the state-of-the-art (among unsupervised methods) mAP of 54.4% that is only 2.4 points lower from the supervised case. We get similarly striking results when we transfer our unsupervised learned features on various other tasks, such as ImageNet classification, PASCAL classification, PASCAL segmentation, and CIFAR-10 classification. The code and models of our paper will be published on: https://github.com/gidariss/FeatureLearningRotNet .

arXiv commentAccepted at ICLR2018. Code and models will be published on: https://github.com/gidariss/FeatureLearningRotNet

References in corpus (9)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Adversarial Feature Learning
• Data-dependent Initializations of Convolutional Neural Networks
• Learning to See by Moving
• Learning Representations for Automatic Colorization
• Colorization as a Proxy Task for Visual Understanding
• Scaling the Scattering Transform: Deep Hybrid Networks
• Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction

Cited by in corpus (131)

• Bootstrap your own latent: A new approach to self-supervised Learning
• Recent advances and clinical applications of deep learning in medical image analysis
• Graph Contrastive Learning with Adaptive Augmentation
• DINOv2: Learning Robust Visual Features without Supervision
• Deep neural network models for computational histopathology: A survey
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• A Survey on Accuracy-oriented Neural Recommendation: From Collaborative Filtering to Information-rich Recommendation
• Multi-task Self-Supervised Learning for Human Activity Detection
• PUFA-GAN: A Frequency-Aware Generative Adversarial Network for 3D Point Cloud Upsampling
• Image Quality Assessment using Contrastive Learning
• Self-supervised learning methods and applications in medical imaging analysis: A survey
• An Overview of Deep Semi-Supervised Learning
• A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts
• A survey on Semi-, Self- and Unsupervised Learning for Image Classification
• SPICE: Semantic Pseudo-labeling for Image Clustering
• Anomaly Detection-Inspired Few-Shot Medical Image Segmentation Through Self-Supervision With Supervoxels
• Self-supervised Domain Adaptation for Computer Vision Tasks
• Unsupervised Point Cloud Representation Learning with Deep Neural Networks: A Survey
• AutoNovel: Automatically Discovering and Learning Novel Visual Categories
• SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding
• Learning Representation for Clustering via Prototype Scattering and Positive Sampling
• Robust Multi-Task Learning and Online Refinement for Spacecraft Pose Estimation across Domain Gap
• Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• Self-Supervised Anomaly Detection in Computer Vision and Beyond: A Survey and Outlook
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• COCOA: Cross Modality Contrastive Learning for Sensor Data
• What makes instance discrimination good for transfer learning?
• Self-supervised deep convolutional neural network for chest X-ray classification
• Towards Label-efficient Automatic Diagnosis and Analysis: A Comprehensive Survey of Advanced Deep Learning-based Weakly-supervised, Semi-supervised and Self-supervised Techniques in Histopathological Image Analysis
• Hierarchical Self-supervised Augmented Knowledge Distillation
• A Systematic Survey of Regularization and Normalization in GANs
• A Review of Predictive and Contrastive Self-supervised Learning for Medical Images
• Federated Contrastive Learning for Volumetric Medical Image Segmentation
• Salient Objects in Clutter
• Neighborhood-Enhanced Supervised Contrastive Learning for Collaborative Filtering
• APANet: Adaptive Prototypes Alignment Network for Few-Shot Semantic Segmentation
• Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences
• Unsupervised Deep Representation Learning and Few-Shot Classification of PolSAR Images
• s-LWSR: Super Lightweight Super-Resolution Network
• Self-Supervised RF Signal Representation Learning for NextG Signal Classification with Deep Learning
• Self-Supervised Nuclei Segmentation in Histopathological Images Using Attention
• SSD: A Unified Framework for Self-Supervised Outlier Detection
• ABC: Auxiliary Balanced Classifier for Class-imbalanced Semi-supervised Learning
• Distilling Knowledge by Mimicking Features
• Roadmap on machine learning glassy dynamics
• LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading
• Contrastive Learning with Stronger Augmentations
• Contrastive Self-supervised Neural Architecture Search
• VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning
• IMU Based Deep Stride Length Estimation With Self-Supervised Learning
• Cycle-Contrast for Self-Supervised Video Representation Learning
• Self-supervised Contrastive Learning of Multi-view Facial Expressions
• Self-Supervised Visual Learning by Variable Playback Speeds Prediction of a Video
• Inherit with Distillation and Evolve with Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory
• Self-Supervised VQ-VAE for One-Shot Music Style Transfer
• A Study of the Generalizability of Self-Supervised Representations
• Parametric Instance Classification for Unsupervised Visual Feature Learning
• Improving colonoscopy lesion classification using semi-supervised deep learning
• Self-supervised visual learning in the low-data regime: a comparative evaluation
• Learning to segment images with classification labels
• Neural Transformation Learning for Deep Anomaly Detection Beyond Images
• Event-guided Multi-patch Network with Self-supervision for Non-uniform Motion Deblurring
• MAD: Self-Supervised Masked Anomaly Detection Task for Multivariate Time Series
• Adaptive Soft Contrastive Learning
• Dual Contrastive Learning for Spatio-temporal Representation
• GaitFormer: Learning Gait Representations with Noisy Multi-Task Learning
• Semi-Supervised Visual Representation Learning for Fashion Compatibility
• Where is the Model Looking At?--Concentrate and Explain the Network Attention
• Contrastive encoder pre-training-based clustered federated learning for heterogeneous data
• Self-Knowledge Distillation based Self-Supervised Learning for Covid-19 Detection from Chest X-Ray Images
• Exploring DeshuffleGANs in Self-Supervised Generative Adversarial Networks
• A Self Supervised StyleGAN for Image Annotation and Classification with Extremely Limited Labels
• Coherent, super resolved radar beamforming using self-supervised learning
• Unsupervised Learning of Video Representations via Dense Trajectory Clustering
• TriBYOL: Triplet BYOL for Self-Supervised Representation Learning
• Beyond Cats and Dogs: Semi-supervised Classification of fuzzy labels with overclustering
• Anatomically aware dual-hop learning for pulmonary embolism detection in CT pulmonary angiograms
• RPT: Toward Transferable Model on Heterogeneous Researcher Data via Pre-Training
• AdaStereo: A Simple and Efficient Approach for Adaptive Stereo Matching
• Learning from Extrinsic and Intrinsic Supervisions for Domain Generalization
• Self-Damaging Contrastive Learning
• Fuzzy Overclustering: Semi-Supervised Classification of Fuzzy Labels with Overclustering and Inverse Cross-Entropy
• Self-supervising Fine-grained Region Similarities for Large-scale Image Localization
• CycAs: Self-supervised Cycle Association for Learning Re-identifiable Descriptions
• Modeling Multiple Views via Implicitly Preserving Global Consistency and Local Complementarity
• Selecting Relevant Features from a Multi-domain Representation for Few-shot Classification
• Self-Supervised Representation Learning for Detection of ACL Tear Injury in Knee MR Videos
• Federated Semi-Supervised Learning for COVID Region Segmentation in Chest CT using Multi-National Data from China, Italy, Japan
• Efficient Self-Supervision using Patch-based Contrastive Learning for Histopathology Image Segmentation
• Unsupervised Learning from Video with Deep Neural Embeddings
• DeshuffleGAN: A Self-Supervised GAN to Improve Structure Learning
• A vector quantized masked autoencoder for audiovisual speech emotion recognition
• Self-Supervised GANs with Label Augmentation
• Self-Supervision, Remote Sensing and Abstraction: Representation Learning Across 3 Million Locations
• CSNNs: Unsupervised, Backpropagation-free Convolutional Neural Networks for Representation Learning
• Revisiting Image Aesthetic Assessment via Self-Supervised Feature Learning
• Reachability Embeddings: Scalable Self-Supervised Representation Learning from Mobility Trajectories for Multimodal Geospatial Computer Vision
• Scalable Self-Supervised Representation Learning from Spatiotemporal Motion Trajectories for Multimodal Computer Vision
• Deep Self-Supervised Hierarchical Clustering for Speaker Diarization
• Representation Learning via Consistent Assignment of Views to Clusters
• INoD: Injected Noise Discriminator for Self-Supervised Representation Learning in Agricultural Fields
• Co-training for On-board Deep Object Detection
• ReSmooth: Detecting and Utilizing OOD Samples when Training with Data Augmentation
• DistillCSE: Distilled Contrastive Learning for Sentence Embeddings
• Contrastive Self-Supervised Learning for Spatio-Temporal Analysis of Lung Ultrasound Videos
• Evolutionary Augmentation Policy Optimization for Self-supervised Learning
• PREGAN: Pose Randomization and Estimation for Weakly Paired Image Style Translation
• Clustering-Based Representation Learning through Output Translation and Its Application to Remote--Sensing Images
• DeepDiffusion: Unsupervised Learning of Retrieval-adapted Representations via Diffusion-based Ranking on Latent Feature Manifold
• Self-supervised Model Based on Masked Autoencoders Advance CT Scans Classification
• Exploring The Spatial Reasoning Ability of Neural Models in Human IQ Tests
• Unsupervised Abnormality Detection Using Heterogeneous Autonomous Systems
• Weakly Supervised Lymph Nodes Segmentation Based on Partial Instance Annotations with Pre-trained Dual-branch Network and Pseudo Label Learning
• On the Challenges of Open World Recognitionunder Shifting Visual Domains
• Shuffle & Divide: Contrastive Learning for Long Text
• One-bit Supervision for Image Classification: Problem, Solution, and Beyond
• Self-supervised Benchmark Lottery on ImageNet: Do Marginal Improvements Translate to Improvements on Similar Datasets?
• Improving Image Clustering through Sample Ranking and Its Application to remote--sensing images
• On the robustness of self-supervised representations for multi-view object classification
• IDEAL: Independent Domain Embedding Augmentation Learning
• Reliable Label Bootstrapping for Semi-Supervised Learning
• Boosting Supervised Learning Performance with Co-training
• Temporal Knowledge Consistency for Unsupervised Visual Representation Learning
• ContraCluster: Learning to Classify without Labels by Contrastive Self-Supervision and Prototype-Based Semi-Supervision
• Entropy-based Active Learning of Graph Neural Network Surrogate Models for Materials Properties
• Visual Probing: Cognitive Framework for Explaining Self-Supervised Image Representations
• SERE: Exploring Feature Self-relation for Self-supervised Transformer
• ScaleNet: An Unsupervised Representation Learning Method for Limited Information
• Clustering augmented Self-Supervised Learning: Anapplication to Land Cover Mapping