Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data

Abstract

Neural machine translation systems have become state-of-the-art approaches for Grammatical Error Correction (GEC) task. In this paper, we propose a copy-augmented architecture for the GEC task by copying the unchanged words from the source sentence to the target sentence. Since the GEC suffers from not having enough labeled training data to achieve high accuracy. We pre-train the copy-augmented architecture with a denoising auto-encoder using the unlabeled One Billion Benchmark and make comparisons between the fully pre-trained model and a partially pre-trained model. It is the first time copying words from the source context and fully pre-training a sequence to sequence model are experimented on the GEC task. Moreover, We add token-level and sentence-level multi-task learning for the GEC task. The evaluation results on the CoNLL-2014 test set show that our approach outperforms all recently published state-of-the-art results by a large margin. The code and pre-trained models are released at https://github.com/zhawe01/fairseq-gec.

Accepted by NAACL 2019 (oral)

References in corpus (7)

• Distributed Representations of Words and Phrases and their Compositionality
• Convolutional Sequence to Sequence Learning
• Get To The Point: Summarization with Pointer-Generator Networks
• Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study
• Data Recombination for Neural Semantic Parsing
• A Nested Attention Neural Hybrid Model for Grammatical Error Correction
• JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction

Cited by in corpus (10)

• Fact-based Text Editing
• Improving Grammatical Error Correction with Machine Translation Pairs
• Parallel Iterative Edit Models for Local Sequence Transduction
• Contextual Text Denoising with Masked Language Models
• Generating similes effortlessly like a Pro: A Style Transfer Approach for Simile Generation
• Parallel Data Augmentation for Formality Style Transfer
• Learning to mirror speaking styles incrementally
• Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction
• CUNI System for the Building Educational Applications 2019 Shared Task: Grammatical Error Correction
• HypoGen: Hyperbole Generation with Commonsense and Counterfactual Knowledge