Continual BERT: Continual Learning for Adaptive Extractive Summarization of COVID-19 Literature
arXiv:2007.03405
Abstract
The scientific community continues to publish an overwhelming amount of new research related to COVID-19 on a daily basis, leading to much literature without little to no attention. To aid the community in understanding the rapidly flowing array of COVID-19 literature, we propose a novel BERT architecture that provides a brief yet original summarization of lengthy papers. The model continually learns on new data in online fashion while minimizing catastrophic forgetting, thus fitting to the need of the community. Benchmark and manual examination of its performance show that the model provide a sound summary of new scientific literature.
6 pages, 3 figures
References in corpus (6)
- Decoupled Weight Decay Regularization
- Overcoming catastrophic forgetting in neural networks
- SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents
- Group Sparse Regularization for Deep Neural Networks
- Progress & Compress: A scalable framework for continual learning
- ERNIE 2.0: A Continual Pre-training Framework for Language Understanding