A Neural Span-Based Continual Named Entity Recognition Model

Abstract

Named Entity Recognition (NER) models capable of Continual Learning (CL) are realistically valuable in areas where entity types continuously increase (e.g., personal assistants). Meanwhile the learning paradigm of NER advances to new patterns such as the span-based methods. However, its potential to CL has not been fully explored. In this paper, we propose SpanKL, a simple yet effective Span-based model with Knowledge distillation (KD) to preserve memories and multi-Label prediction to prevent conflicts in CL-NER. Unlike prior sequence labeling approaches, the inherently independent modeling in span and entity level with the designed coherent optimization on SpanKL promotes its learning at each incremental step and mitigates the forgetting. Experiments on synthetic CL datasets derived from OntoNotes and Few-NERD show that SpanKL significantly outperforms previous SoTA in many aspects, and obtains the smallest gap from CL to the upper bound revealing its high practiced value. The code is available at https://github.com/Qznan/SpanKL.

arXiv commentAccepted by AAAI'23 (Update to official format)

References in corpus (13)

• Distilling the Knowledge in a Neural Network
• Decoupled Weight Decay Regularization
• Overcoming catastrophic forgetting in neural networks
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• A continual learning survey: Defying forgetting in classification tasks
• Three scenarios for continual learning
• Overcoming catastrophic forgetting with hard attention to the task
• Reinforced Continual Learning
• Class-incremental learning: survey and performance evaluation on image classification
• Neural Topic Modeling with Continual Lifelong Learning
• Transfer Learning for Sequence Labeling Using Source Model and Target Data
• Structural Knowledge Distillation: Tractably Distilling Information for Structured Predictor
• SpanNER: Named Entity Re-/Recognition as Span Prediction