A Survey on Transfer Learning in Natural Language Processing

Abstract

Deep learning models usually require a huge amount of data. However, these large datasets are not always attainable. This is common in many challenging NLP tasks. Consider Neural Machine Translation, for instance, where curating such large datasets may not be possible specially for low resource languages. Another limitation of deep learning models is the demand for huge computing resources. These obstacles motivate research to question the possibility of knowledge transfer using large trained models. The demand for transfer learning is increasing as many large models are emerging. In this survey, we feature the recent transfer learning advances in the field of NLP. We also provide a taxonomy for categorizing different transfer learning approaches from the literature.

References in corpus (21)

• Distilling the Knowledge in a Neural Network
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Deep Domain Confusion: Maximizing for Domain Invariance
• Convolutional Neural Networks for Sentence Classification
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
• Dynamic Coattention Networks For Question Answering
• Regularizing and Optimizing LSTM Language Models
• A large annotated corpus for learning natural language inference
• Machine Comprehension Using Match-LSTM and Answer Pointer
• Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
• RACE: Large-scale ReAding Comprehension Dataset From Examinations
• Transfer Learning for Named-Entity Recognition with Neural Networks
• Knowledge Adaptation: Teaching to Adapt
• Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach
• Evolution of transfer learning in natural language processing
• Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary
• Adversarial Domain Adaptation for Duplicate Question Detection

Cited by in corpus (5)

• Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization
• Deep transfer learning for intrusion detection in industrial control networks: A comprehensive review
• Literature review on vulnerability detection using NLP technology
• FAQ-Gen: An automated system to generate domain-specific FAQs to aid content comprehension
• Privacy enabled Financial Text Classification using Differential Privacy and Federated Learning