Segmental Recurrent Neural Networks

Abstract

We introduce segmental recurrent neural networks (SRNNs) which define, given an input sequence, a joint probability distribution over segmentations of the input and labelings of the segments. Representations of the input segments (i.e., contiguous subsequences of the input) are computed by encoding their constituent tokens using bidirectional recurrent neural nets, and these "segment embeddings" are used to define compatibility scores with output labels. These local compatibility scores are integrated using a global semi-Markov conditional random field. Both fully supervised training -- in which segment boundaries and labels are observed -- as well as partially supervised training -- in which segment boundaries are latent -- are straightforward. Experiments on handwriting recognition and joint Chinese word segmentation/POS tagging show that, compared to models that do not explicitly represent segments such as BIO tagging schemes and connectionist temporal classification (CTC), SRNNs obtain substantially higher accuracies.

10 pages, published as a conference paper at ICLR 2016

Cited by in corpus (32)

• DyNet: The Dynamic Neural Network Toolkit
• Character-based Joint Segmentation and POS Tagging for Chinese using Bidirectional RNN-CRF
• Neural Speed Reading via Skim-RNN
• Segmental Recurrent Neural Networks for End-to-end Speech Recognition
• Sequence Modeling via Segmentations
• A Tutorial on Deep Latent Variable Models of Natural Language
• Monotonic Chunkwise Attention
• End-to-End Neural Segmental Models for Speech Recognition
• Exploring Segment Representations for Neural Segmentation Models
• Neural Segmental Hypergraphs for Overlapping Mention Recognition
• Bayesian Online Prediction of Change Points
• Learning Structured Natural Language Representations for Semantic Parsing
• Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling
• Multitask Learning with CTC and Segmental CRF for Speech Recognition
• Time Perception Machine: Temporal Point Processes for the When, Where and What of Activity Prediction
• Neural Phrase-to-Phrase Machine Translation
• Online Segment to Segment Neural Transduction
• Learning Neural Templates for Text Generation
• Syntactic Scaffolds for Semantic Structures
• Learning Joint Semantic Parsers from Disjoint Data
• A Span Selection Model for Semantic Role Labeling
• Sequence Prediction with Neural Segmental Models
• Neural Semi-Markov Conditional Random Fields for Robust Character-Based Part-of-Speech Tagging
• Learning Explicit and Implicit Structures for Targeted Sentiment Analysis
• Low-pass Recurrent Neural Networks - A memory architecture for longer-term correlation discovery
• PaLM: A Hybrid Parser and Language Model
• Latent-Variable Generative Models for Data-Efficient Text Classification
• Joint Semantic Synthesis and Morphological Analysis of the Derived Word
• Differentiable Segmentation of Sequences
• Whole-Word Segmental Speech Recognition with Acoustic Word Embeddings
• Segmenting Natural Language Sentences via Lexical Unit Analysis
• Sequence-to-Sequence Learning with Latent Neural Grammars