Neural Named Entity Recognition from Subword Units
arXiv:1808.07364 · doi:10.21437/Interspeech.2019-1305
Abstract
Named entity recognition (NER) is a vital task in spoken language understanding, which aims to identify mentions of named entities in text e.g., from transcribed speech. Existing neural models for NER rely mostly on dedicated word-level representations, which suffer from two main shortcomings. First, the vocabulary size is large, yielding large memory requirements and training time. Second, these models are not able to learn morphological or phonological representations. To remedy the above shortcomings, we adopt a neural solution based on bidirectional LSTMs and conditional random fields, where we rely on subword units, namely characters, phonemes, and bytes. For each word in an utterance, our model learns a representation from each of the subword units. We conducted experiments in a real-world large-scale setting for the use case of a voice-controlled device covering four languages with up to 5.5M utterances per language. Our experiments show that (1) with increasing training data, performance of models trained solely on subword units becomes closer to that of models with dedicated word-level embeddings (91.35 vs 93.92 F1 for English), while using a much smaller vocabulary size (332 vs 74K), (2) subword units enhance models with dedicated word-level embeddings, and (3) combining different subword units improves performance.
arXiv comment5 pages, INTERSPEECH 2019
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- Improving neural networks by preventing co-adaptation of feature detectors
- Natural Language Processing (almost) from Scratch
- Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition
- A Survey on Recent Advances in Named Entity Recognition from Deep Learning models
- Multi-Task Cross-Lingual Sequence Tagging from Scratch