Representation Stability as a Regularizer for Improved Text Analytics Transfer Learning

Abstract

Although neural networks are well suited for sequential transfer learning tasks, the catastrophic forgetting problem hinders proper integration of prior knowledge. In this work, we propose a solution to this problem by using a multi-task objective based on the idea of distillation and a mechanism that directly penalizes forgetting at the shared representation layer during the knowledge integration phase of training. We demonstrate our approach on a Twitter domain sentiment analysis task with sequential knowledge transfer from four related tasks. We show that our technique outperforms networks fine-tuned to the target task. Additionally, we show both through empirical evidence and examples that it does not forget useful knowledge from the source task that is forgotten during standard fine-tuning. Surprisingly, we find that first distilling a human made rule based sentiment engine into a recurrent neural network and then integrating the knowledge with the target task data leads to a substantial gain in generalization performance. Our experiments demonstrate the power of multi-source transfer techniques in practical text analytics problems when paired with distillation. In particular, for the SemEval 2016 Task 4 Subtask A (Nakov et al., 2016) dataset we surpass the state of the art established during the competition with a comparatively simple model architecture that is not even competitive when trained on only the labeled task specific data.

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• FitNets: Hints for Thin Deep Nets
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• Harnessing Deep Neural Networks with Logic Rules
• Net2Net: Accelerating Learning via Knowledge Transfer
• Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews

Cited by in corpus (5)

• Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
• Learning Abstract Options
• Refining Language Models with Compositional Explanations
• Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models
• Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension