Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction
arXiv:2005.00987
Abstract
This paper investigates how to effectively incorporate a pre-trained masked language model (MLM), such as BERT, into an encoder-decoder (EncDec) model for grammatical error correction (GEC). The answer to this question is not as straightforward as one might expect because the previous common methods for incorporating a MLM into an EncDec model have potential drawbacks when applied to GEC. For example, the distribution of the inputs to a GEC model can be considerably different (erroneous, clumsy, etc.) from that of the corpora used for pre-training MLMs; however, this issue is not addressed in the previous methods. Our experiments show that our proposed method, where we first fine-tune a MLM with a given GEC corpus and then use the output of the fine-tuned MLM as additional features in the GEC model, maximizes the benefit of the MLM. The best-performing model achieves state-of-the-art performances on the BEA-2019 and CoNLL-2014 benchmarks. Our code is publicly available at: https://github.com/kanekomasahiro/bert-gec.
Accepted as a short paper to the 58th Annual Conference of the Association for Computational Linguistics (ACL-2020)
References in corpus (6)
- Pre-trained Models for Natural Language Processing: A Survey
- Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
- Incorporating BERT into Neural Machine Translation
- Compositional Sequence Labeling Models for Error Detection in Learner Writing
- Context is Key: Grammatical Error Detection with Contextual Word Representations
- Acquiring Knowledge from Pre-trained Model to Neural Machine Translation
Cited by in corpus (4)
- Generating Bug-Fixes Using Pretrained Transformers
- Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction
- Few-Shot Domain Adaptation for Grammatical Error Correction via Meta-Learning
- A Self-Refinement Strategy for Noise Reduction in Grammatical Error Correction