A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks

Abstract

Deep neural networks are typically trained under a supervised learning framework where a model learns a single task using labeled data. Instead of relying solely on labeled data, practitioners can harness unlabeled or related data to improve model performance, which is often more accessible and ubiquitous. Self-supervised pre-training for transfer learning is becoming an increasingly popular technique to improve state-of-the-art results using unlabeled data. It involves first pre-training a model on a large amount of unlabeled data, then adapting the model to target tasks of interest. In this review, we survey self-supervised learning methods and their applications within the sequential transfer learning framework. We provide an overview of the taxonomy for self-supervised learning and transfer learning, and highlight some prominent methods for designing pre-training tasks across different domains. Finally, we discuss recent trends and suggest areas for future investigation.

References in corpus (29)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• A Simple Framework for Contrastive Learning of Visual Representations
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Language Models are Few-Shot Learners
• Neural Discrete Representation Learning
• An Overview of Multi-Task Learning in Deep Neural Networks
• Deep Visual Domain Adaptation: A Survey
• Unsupervised Learning of Video Representations using LSTMs
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• ERNIE: Enhanced Representation through Knowledge Integration
• Adversarial Feature Learning
• Semi-Supervised Learning with Ladder Networks
• Semi-supervised Sequence Learning
• A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
• A survey on Semi-, Self- and Unsupervised Learning for Image Classification
• Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
• ReZero is All You Need: Fast Convergence at Large Depth
• k-Sparse Autoencoders
• Jukebox: A Generative Model for Music
• A Mutual Information Maximization Perspective of Language Representation Learning
• Selfie: Self-supervised Pretraining for Image Embedding
• Unsupervised Translation of Programming Languages
• Automatic Shortcut Removal for Self-Supervised Representation Learning
• Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• LakhNES: Improving multi-instrumental music generation with cross-domain pre-training
• Deep Automated Multi-task Learning

Cited by in corpus (5)

• Self-supervised learning methods and applications in medical imaging analysis: A survey
• Text-Conditioned Transformer for Automatic Pronunciation Error Detection
• Techniques for Symbol Grounding with SATNet
• Re-entry Prediction for Online Conversations via Self-Supervised Learning
• Vision-Guided Forecasting -- Visual Context for Multi-Horizon Time Series Forecasting