Neural Language Correction with Character-Based Attention

Abstract

Natural language correction has the potential to help language learners improve their writing skills. While approaches with separate classifiers for different error types have high precision, they do not flexibly handle errors such as redundancy or non-idiomatic phrasing. On the other hand, word and phrase-based machine translation methods are not designed to cope with orthographic errors, and have recently been outpaced by neural models. Motivated by these issues, we present a neural network-based approach to language correction. The core component of our method is an encoder-decoder recurrent neural network with an attention mechanism. By operating at the character level, the network avoids the problem of out-of-vocabulary words. We illustrate the flexibility of our approach on dataset of noisy, user-generated text collected from an English learner forum. When combined with a language model, our method achieves a state-of-the-art $F_{0.5}$-score on the CoNLL 2014 Shared Task. We further demonstrate that training the network on additional data with synthesized errors can improve performance.

10 pages

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Listen, Attend and Spell
• Character-based Neural Machine Translation
• The CoNLL-2013 Shared Task on Grammatical Error Correction
• Addressing the Rare Word Problem in Neural Machine Translation

Cited by in corpus (27)

• Grammatical Error Correction: A Survey of the State of the Art
• Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study
• Neural Networks for Text Correction and Completion in Keyboard Decoding
• Tree-Transformer: A Transformer-Based Method for Correction of Tree-Structured Data
• A Comprehensive Survey of Grammar Error Correction
• From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning
• A Nested Attention Neural Hybrid Model for Grammatical Error Correction
• Character-Word LSTM Language Models
• Referenceless Quality Estimation for Natural Language Generation
• Inherent Biases in Reference based Evaluation for Grammatical Error Correction and Text Simplification
• Still not there? Comparing Traditional Sequence-to-Sequence Models to Encoder-Decoder Neural Networks on Monotone String Translation Tasks
• Parallel Iterative Edit Models for Local Sequence Transduction
• Investigating an Effective Character-level Embedding in Korean Sentence Classification
• Utilizing Character and Word Embeddings for Text Normalization with Sequence-to-Sequence Models
• Denoising based Sequence-to-Sequence Pre-training for Text Generation
• Joint Contextual Modeling for ASR Correction and Language Understanding
• Towards the Unseen: Iterative Text Recognition by Distilling from Errors
• A spelling correction model for end-to-end speech recognition
• Generative Question Refinement with Deep Reinforcement Learning in Retrieval-based QA System
• LM-Critic: Language Models for Unsupervised Grammatical Error Correction
• Application of Seq2Seq Models on Code Correction
• Hierarchical Attention Transformer Architecture For Syntactic Spell Correction
• KNPTC: Knowledge and Neural Machine Translation Powered Chinese Pinyin Typo Correction
• Graph-based Filtering of Out-of-Vocabulary Words for Encoder-Decoder Models
• Hierarchical Character Tagger for Short Text Spelling Error Correction
• Corpora Generation for Grammatical Error Correction
• Neural Transductive Learning and Beyond: Morphological Generation in the Minimal-Resource Setting