Compositional Morphology for Word Representations and Language Modelling

Abstract

This paper presents a scalable method for integrating compositional morphological representations into a vector-based probabilistic language model. Our approach is evaluated in the context of log-bilinear language models, rendered suitably efficient for implementation inside a machine translation decoder by factoring the vocabulary. We perform both intrinsic and extrinsic evaluations, presenting results on a range of languages which demonstrate that our model learns morphological representations that both perform well on word similarity tasks and lead to substantial reductions in perplexity. When used for translation into morphologically rich languages with large vocabularies, our models obtain improvements of up to 1.2 BLEU points relative to a baseline system using back-off n-gram models.

Proceedings of the 31st International Conference on Machine Learning (ICML)

References in corpus (2)

• Efficient Estimation of Word Representations in Vector Space
• A Fast and Simple Algorithm for Training Neural Probabilistic Language Models

Cited by in corpus (16)

• Character-Aware Neural Language Models
• Natural Language Processing Advancements By Deep Learning: A Survey
• LightRNN: Memory and Computation-Efficient Recurrent Neural Networks
• Confusion2Vec: Towards Enriching Vector Space Word Representations with Representational Ambiguities
• A Survey On Neural Word Embeddings
• Probabilistic Modelling of Morphologically Rich Languages
• Learning Latent Local Conversation Modes for Predicting Community Endorsement in Online Discussions
• The boundaries of meaning: a case study in neural machine translation
• Evaluation of Morphological Embeddings for the Russian Language
• Evaluation of Morphological Embeddings for English and Russian Languages
• Effective Subword Segmentation for Text Comprehension
• Using Interlinear Glosses as Pivot in Low-Resource Multilingual Machine Translation
• Gating Mechanisms for Combining Character and Word-level Word Representations: An Empirical Study
• Comparison of Turkish Word Representations Trained on Different Morphological Forms
• Stem-driven Language Models for Morphologically Rich Languages
• Revisiting Neural Language Modelling with Syllables