Utilizing Character and Word Embeddings for Text Normalization with Sequence-to-Sequence Models
arXiv:1809.01534
Abstract
Text normalization is an important enabling technology for several NLP tasks. Recently, neural-network-based approaches have outperformed well-established models in this task. However, in languages other than English, there has been little exploration in this direction. Both the scarcity of annotated data and the complexity of the language increase the difficulty of the problem. To address these challenges, we use a sequence-to-sequence model with character-based attention, which in addition to its self-learned character embeddings, uses word embeddings pre-trained with an approach that also models subword information. This provides the neural model with access to more linguistic information especially suitable for text normalization, without large parallel corpora. We show that providing the model with word-level features bridges the gap for the neural network approach to achieve a state-of-the-art F1 score on a standard Arabic language correction shared task dataset.
arXiv commentAccepted in EMNLP 2018
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- Neural Machine Translation by Jointly Learning to Align and Translate
- Sequence to Sequence Learning with Neural Networks
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- Effective Approaches to Attention-based Neural Machine Translation
- Enriching Word Vectors with Subword Information
- Neural Language Correction with Character-Based Attention
- Attention-based Encoder-Decoder Networks for Spelling and Grammatical Error Correction