Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey

Abstract

Large-scale labeled data are generally required to train deep neural networks in order to obtain better performance in visual feature learning from images or videos for computer vision applications. To avoid extensive cost of collecting and annotating large-scale datasets, as a subset of unsupervised learning methods, self-supervised learning methods are proposed to learn general image and video features from large-scale unlabeled data without using any human-annotated labels. This paper provides an extensive review of deep learning-based self-supervised general visual feature learning methods from images or videos. First, the motivation, general pipeline, and terminologies of this field are described. Then the common deep neural network architectures that used for self-supervised learning are summarized. Next, the main components and evaluation metrics of self-supervised learning methods are reviewed followed by the commonly used image and video datasets and the existing self-supervised visual feature learning methods. Finally, quantitative performance comparisons of the reviewed methods on benchmark datasets are summarized and discussed for both image and video feature learning. At last, this paper is concluded and lists a set of promising future directions for self-supervised visual feature learning.

References in corpus (31)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Two-Stream Convolutional Networks for Action Recognition in Videos
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• The Kinetics Human Action Video Dataset
• Unsupervised Learning of Video Representations using LSTMs
• Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
• Understanding deep learning requires rethinking generalization
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Adversarial Feature Learning
• Spatiotemporal Residual Networks for Video Action Recognition
• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• Towards Good Practices for Very Deep Two-Stream ConvNets
• WebVision Database: Visual Learning and Understanding from Web Data
• Semantic Understanding of Scenes through the ADE20K Dataset
• GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
• Places: An Image Database for Deep Scene Understanding
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Improvements to context based self-supervised learning
• Stochastic Variational Video Prediction
• Real-Time User-Guided Image Colorization with Learned Deep Priors
• Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles
• Learning Image Representations by Completing Damaged Jigsaw Puzzles
• Pose from Action: Unsupervised Learning of Pose Features based on Motion
• Deep unsupervised learning through spatial contrasting
• Self-Supervised Feature Learning by Learning to Spot Artifacts
• Self-Supervised GAN to Counter Forgetting
• Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition

Cited by in corpus (47)

• Transformers in Vision: A Survey
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• An Overview of Deep Semi-Supervised Learning
• RODNet: A Real-Time Radar Object Detection Network Cross-Supervised by Camera-Radar Fused Object 3D Localization
• Self-supervised Domain Adaptation for Computer Vision Tasks
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• AeroRIT: A New Scene for Hyperspectral Image Analysis
• Contrastive Learning with Adversarial Examples
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis
• LogME: Practical Assessment of Pre-trained Models for Transfer Learning
• Self-Supervised Similarity Learning for Digital Pathology
• Is it enough to optimize CNN architectures on ImageNet?
• Socially Enhanced Situation Awareness from Microblogs using Artificial Intelligence: A Survey
• Simple Unsupervised Multi-Object Tracking
• Towards Shape Biased Unsupervised Representation Learning for Domain Generalization
• A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks
• Deep Comprehensive Correlation Mining for Image Clustering
• PredNet and Predictive Coding: A Critical Review
• Accurate and Robust Pulmonary Nodule Detection by 3D Feature Pyramid Network with Self-supervised Feature Learning
• SelFlow: Self-Supervised Learning of Optical Flow
• Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
• Proposal Learning for Semi-Supervised Object Detection
• Self-Supervised Features Improve Open-World Learning
• Exploring Self-Supervised Regularization for Supervised and Semi-Supervised Learning
• How Useful is Self-Supervised Pretraining for Visual Tasks?
• S3VAE: Self-Supervised Sequential VAE for Representation Disentanglement and Data Generation
• Unsupervised Feature Learning for Point Cloud by Contrasting and Clustering With Graph Convolutional Neural Network
• Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training
• Self-Supervised Multi-View Learning via Auto-Encoding 3D Transformations
• Self-supervised Feature Learning by Cross-modality and Cross-view Correspondences
• VideoSSL: Semi-Supervised Learning for Video Classification
• Strength from Weakness: Fast Learning Using Weak Supervision
• Towards a Hypothesis on Visual Transformation based Self-Supervision
• SleepPriorCL: Contrastive Representation Learning with Prior Knowledge-based Positive Mining and Adaptive Temperature for Sleep Staging
• Self-supervised visual feature learning with curriculum
• Self-supervised pre-training with acoustic configurations for replay spoofing detection
• Region-of-interest guided Supervoxel Inpainting for Self-supervision
• Similarity Guided Deep Face Image Retrieval
• Knowledge as Invariance -- History and Perspectives of Knowledge-augmented Machine Learning
• Information based Deep Clustering: An experimental study
• A Multi-view Perspective of Self-supervised Learning
• Attention-Based Self-Supervised Feature Learning for Security Data
• Exploit Clues from Views: Self-Supervised and Regularized Learning for Multiview Object Recognition
• Longitudinal Self-Supervised Learning