Continual Learning for Sentence Representations Using Conceptors
arXiv:1904.09187
Abstract
Distributed representations of sentences have become ubiquitous in natural language processing tasks. In this paper, we consider a continual learning scenario for sentence representations: Given a sequence of corpora, we aim to optimize the sentence encoder with respect to the new corpus while maintaining its accuracy on the old corpora. To address this problem, we propose to initialize sentence encoders with the help of corpus-independent features, and then sequentially update sentence encoders using Boolean operations of conceptor matrices to learn corpus-dependent features. We evaluate our approach on semantic textual similarity tasks and show that our proposed sentence encoder can continually learn features from new corpora while retaining its competence on previously encountered corpora.
Accepted by NAACL-2019
References in corpus (7)
- Universal Sentence Encoder
- Continual Learning Through Synaptic Intelligence
- Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
- SentEval: An Evaluation Toolkit for Universal Sentence Representations
- Controlling Recurrent Neural Networks by Conceptors
- Toward Continual Learning for Conversational Agents
- The Rapidly Changing Landscape of Conversational Agents
Cited by in corpus (4)
- Exploring the relationship between team institutional composition and novelty in academic papers based on fine-grained knowledge entities
- Lifelong Language Knowledge Distillation
- Dynamically Addressing Unseen Rumor via Continual Learning
- Online Continual Learning in Image Classification: An Empirical Survey