Neural Skill Transfer from Supervised Language Tasks to Reading Comprehension
arXiv:1711.03754
Abstract
Reading comprehension is a challenging task in natural language processing and requires a set of skills to be solved. While current approaches focus on solving the task as a whole, in this paper, we propose to use a neural network `skill' transfer approach. We transfer knowledge from several lower-level language tasks (skills) including textual entailment, named entity recognition, paraphrase detection and question type classification into the reading comprehension model. We conduct an empirical evaluation and show that transferring language skill knowledge leads to significant improvements for the task with much fewer steps compared to the baseline model. We also show that the skill transfer approach is effective even with small amounts of training data. Another finding of this work is that using token-wise deep label supervision for text classification improves the performance of transfer learning.
References in corpus (12)
- An Overview of Multi-Task Learning in Deep Neural Networks
- Learning to Diagnose with LSTM Recurrent Neural Networks
- A large annotated corpus for learning natural language inference
- ReasoNet: Learning to Stop Reading in Machine Comprehension
- One Model To Learn Them All
- A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
- Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
- MEMEN: Multi-layer Embedding with Memory Networks for Machine Comprehension
- DisSent: Sentence Representation Learning from Explicit Discourse Relations
- Embracing data abundance: BookTest Dataset for Reading Comprehension
- Making Neural QA as Simple as Possible but not Simpler
- Who did What: A Large-Scale Person-Centered Cloze Dataset