Segmenting Natural Language Sentences via Lexical Unit Analysis

Abstract

In this work, we present Lexical Unit Analysis (LUA), a framework for general sequence segmentation tasks. Given a natural language sentence, LUA scores all the valid segmentation candidates and utilizes dynamic programming (DP) to extract the maximum scoring one. LUA enjoys a number of appealing properties such as inherently guaranteeing the predicted segmentation to be valid and facilitating globally optimal training and inference. Besides, the practical time complexity of LUA can be reduced to linear time, which is very efficient. We have conducted extensive experiments on 5 tasks, including syntactic chunking, named entity recognition (NER), slot filling, Chinese word segmentation, and Chinese part-of-speech (POS) tagging, across 15 datasets. Our models have achieved the state-of-the-art performances on 13 of them. The results also show that the F1 score of identifying long-length segments is notably improved.

References in corpus (5)

• Bidirectional Attention Flow for Machine Comprehension
• Segmental Recurrent Neural Networks
• Character-based Joint Segmentation and POS Tagging for Chinese using Bidirectional RNN-CRF
• TexSmart: A Text Understanding System for Fine-Grained NER and Enhanced Semantic Analysis
• Named Entity Recognition as Dependency Parsing