Unsupervised Discovery of Morphemes

Abstract

We present two methods for unsupervised segmentation of words into morpheme-like units. The model utilized is especially suited for languages with a rich morphology, such as Finnish. The first method is based on the Minimum Description Length (MDL) principle and works online. In the second method, Maximum Likelihood (ML) optimization is used. The quality of the segmentations is measured using an evaluation method that compares the segmentations produced to an existing morphological analysis. Experiments on both Finnish and English corpora show that the presented methods perform well compared to a current state-of-the-art system.

10 pages, to appear in Proceedings of Morphological and Phonological Learning Workshop of ACL'02

References in corpus (2)

• The Unsupervised Acquisition of a Lexicon from Continuous Speech
• Linguistic Structure as Composition and Perturbation

Cited by in corpus (10)

• Challenges of language technologies for the indigenous languages of the Americas
• Acquisition of morphological families and derivational series from a machine readable dictionary
• Turkish PoS Tagging by Reducing Sparsity with Morpheme Tags in Small Datasets
• Paraphrase Detection on Noisy Subtitles in Six Languages
• Neural Multi-Source Morphological Reinflection
• Building Morphological Chains for Agglutinative Languages
• Cognate-aware morphological segmentation for multilingual neural translation
• Incremental Learning for Fully Unsupervised Word Segmentation Using Penalized Likelihood and Model Selection
• MORSE: Semantic-ally Drive-n MORpheme SEgment-er
• Stem-driven Language Models for Morphologically Rich Languages