Sequence Modeling via Segmentations

Abstract

Segmental structure is a common pattern in many types of sequences such as phrases in human languages. In this paper, we present a probabilistic model for sequences via their segmentations. The probability of a segmented sequence is calculated as the product of the probabilities of all its segments, where each segment is modeled using existing tools such as recurrent neural networks. Since the segmentation of a sequence is usually unknown in advance, we sum over all valid segmentations to obtain the final probability for the sequence. An efficient dynamic programming algorithm is developed for forward and backward computations without resorting to any approximation. We demonstrate our approach on text segmentation and speech recognition tasks. In addition to quantitative results, we also show that our approach can discover meaningful segments in their respective application contexts.

arXiv commentrecurrent neural networks, dynamic programming, structured prediction

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Sequence Transduction with Recurrent Neural Networks
• Segmental Recurrent Neural Networks
• Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks
• Online Segment to Segment Neural Transduction

Cited by in corpus (9)

• Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
• Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction
• CompILE: Compositional Imitation Learning and Execution
• Phrase-Based Attentions
• Learning Neural Templates for Text Generation
• SelfSeg: A Self-supervised Sub-word Segmentation Method for Neural Machine Translation
• Learning to Discover, Ground and Use Words with Segmental Neural Language Models
• Subword Language Model for Query Auto-Completion