Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks

Abstract

Recent papers have shown that neural networks obtain state-of-the-art performance on several different sequence tagging tasks. One appealing property of such systems is their generality, as excellent performance can be achieved with a unified architecture and without task-specific feature engineering. However, it is unclear if such systems can be used for tasks without large amounts of training data. In this paper we explore the problem of transfer learning for neural sequence taggers, where a source task with plentiful annotations (e.g., POS tagging on Penn Treebank) is used to improve performance on a target task with fewer available annotations (e.g., POS tagging for microblogs). We examine the effects of transfer learning for deep hierarchical recurrent networks across domains, applications, and languages, and show that significant improvement can often be obtained. These improvements lead to improvements over the current state-of-the-art on several well-studied tasks.

Accepted as a conference paper at ICLR 2017. This is an extended version of the original paper (https://arxiv.org/abs/1603.06270). The original paper proposes a new architecture, while this version focuses on transfer learning for a general model class

Cited by in corpus (44)

• A Survey on Recent Advances in Named Entity Recognition from Deep Learning models
• Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields
• Multi-Task Learning for Sequence Tagging: An Empirical Study
• Neural ranking models for document retrieval
• AdaRNN: Adaptive Learning and Forecasting of Time Series
• Position-Aware Self-Attention based Neural Sequence Labeling
• Learning to Selectively Transfer: Reinforced Transfer Learning for Deep Text Matching
• CrossNER: Evaluating Cross-Domain Named Entity Recognition
• Cross-lingual Argumentation Mining: Machine Translation (and a bit of Projection) is All You Need!
• Neural Aspect and Opinion Term Extraction with Mined Rules as Weak Supervision
• A Survey on Recent Advances in Sequence Labeling from Deep Learning Models
• Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog
• Cross-Domain Contract Element Extraction with a Bi-directional Feedback Clause-Element Relation Network
• A Comprehensive Analysis of Information Leakage in Deep Transfer Learning
• Transfer Learning for Context-Aware Question Matching in Information-seeking Conversations in E-commerce
• GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labeling
• Extracting Symptoms and their Status from Clinical Conversations
• Learning to Infer Entities, Properties and their Relations from Clinical Conversations
• End-to-End Slot Alignment and Recognition for Cross-Lingual NLU
• Enhanced Meta-Learning for Cross-lingual Named Entity Recognition with Minimal Resources
• Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling
• Zero-Resource Cross-Domain Named Entity Recognition
• Latent-Optimized Adversarial Neural Transfer for Sarcasm Detection
• Reinforced Iterative Knowledge Distillation for Cross-Lingual Named Entity Recognition
• Label-Agnostic Sequence Labeling by Copying Nearest Neighbors
• Low-Resource Name Tagging Learned with Weakly Labeled Data
• A Rigorous Study on Named Entity Recognition: Can Fine-tuning Pretrained Model Lead to the Promised Land?
• Neural Adaptation Layers for Cross-domain Named Entity Recognition
• Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains
• AdaTag: Multi-Attribute Value Extraction from Product Profiles with Adaptive Decoding
• Collecting Indicators of Compromise from Unstructured Text of Cybersecurity Articles using Neural-Based Sequence Labelling
• Multi-Task Learning with Contextualized Word Representations for Extented Named Entity Recognition
• Modelling Domain Relationships for Transfer Learning on Retrieval-based Question Answering Systems in E-commerce
• Global Attention for Name Tagging
• EasyTransfer -- A Simple and Scalable Deep Transfer Learning Platform for NLP Applications
• Soft Layer Selection with Meta-Learning for Zero-Shot Cross-Lingual Transfer
• Data Annealing for Informal Language Understanding Tasks
• Coarse-to-Fine Pre-training for Named Entity Recognition
• Cross-Lingual Transfer for Distantly Supervised and Low-resources Indonesian NER
• Variational Sequential Labelers for Semi-Supervised Learning
• Recognising Biomedical Names: Challenges and Solutions
• A Survey on Green Deep Learning
• Multi-task Learning Based Neural Bridging Reference Resolution
• Free the Plural: Unrestricted Split-Antecedent Anaphora Resolution