A Review of Deep Transfer Learning and Recent Advancements

Abstract

Deep learning has been the answer to many machine learning problems during the past two decades. However, it comes with two major constraints: dependency on extensive labeled data and training costs. Transfer learning in deep learning, known as Deep Transfer Learning (DTL), attempts to reduce such dependency and costs by reusing an obtained knowledge from a source data/task in training on a target data/task. Most applied DTL techniques are network/model-based approaches. These methods reduce the dependency of deep learning models on extensive training data and drastically decrease training costs. As a result, researchers detected Covid-19 infection on chest X-Rays with high accuracy at the beginning of the pandemic with minimal data using DTL techniques. Also, the training cost reduction makes DTL viable on edge devices with limited resources. Like any new advancement, DTL methods have their own limitations, and a successful transfer depends on some adjustments for different scenarios. In this paper, we review the definition and taxonomy of deep transfer learning and well-known methods. Then we investigate the DTL approaches by reviewing recent applied DTL techniques in the past five years. Further, we review some experimental analyses of DTLs to learn the best practice for applying DTL in different scenarios. Moreover, the limitations of DTLs (catastrophic forgetting dilemma and overly biased pre-trained models) are discussed, along with possible solutions and research trends.

18 pages, 2 figures, 1 table

Cited by in corpus (9)

• Accelerating high order discontinuous Galerkin solvers using neural networks: 3D compressible Navier-Stokes equations
• Transfer learning with generative models for object detection on limited datasets
• On the use of neural networks for the structural characterization of polymeric porous materials
• A Classifier-Based Approach to Multi-Class Anomaly Detection for Astronomical Transients
• Mobile Traffic Prediction at the Edge Through Distributed and Deep Transfer Learning
• Bias correction of wind power forecasts with SCADA data and continuous learning
• Robust and Explainable Framework to Address Data Scarcity in Diagnostic Imaging
• Bridging Data Gaps in Healthcare: A Scoping Review of Transfer Learning in Biomedical Data Analysis
• Transfer learning of many-body electronic correlation entropy from local measurements