FineText: Text Classification via Attention-based Language Model Fine-tuning
arXiv:1910.11959
Abstract
Training deep neural networks from scratch on natural language processing (NLP) tasks requires significant amount of manually labeled text corpus and substantial time to converge, which usually cannot be satisfied by the customers. In this paper, we aim to develop an effective transfer learning algorithm by fine-tuning a pre-trained language model. The goal is to provide expressive and convenient-to-use feature extractors for downstream NLP tasks, and achieve improvement in terms of accuracy, data efficiency, and generalization to new domains. Therefore, we propose an attention-based fine-tuning algorithm that automatically selects relevant contextualized features from the pre-trained language model and uses those features on downstream text classification tasks. We test our methods on six widely-used benchmarking datasets, and achieve new state-of-the-art performance on all of them. Moreover, we then introduce an alternative multi-task learning approach, which is an end-to-end algorithm given the pre-trained model. By doing multi-task learning, one can largely reduce the total training time by trading off some classification accuracy.
References in corpus (9)
- A Deep Reinforced Model for Abstractive Summarization
- Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
- Pointer Sentinel Mixture Models
- Regularizing and Optimizing LSTM Language Models
- Learning to Generate Reviews and Discovering Sentiment
- Selective Encoding for Abstractive Sentence Summarization
- Long Short-Term Memory-Networks for Machine Reading
- DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
- Convolutional Neural Networks for Text Categorization: Shallow Word-level vs. Deep Character-level