Document Context Language Models
arXiv:1511.03962
Abstract
Text documents are structured on multiple levels of detail: individual words are related by syntax, but larger units of text are related by discourse structure. Existing language models generally fail to account for discourse structure, but it is crucial if we are to have language models that reward coherence and generate coherent texts. We present and empirically evaluate a set of multi-level recurrent neural network language models, called Document-Context Language Models (DCLM), which incorporate contextual information both within and beyond the sentence. In comparison with word-level recurrent neural network language models, the DCLM models obtain slightly better predictive likelihoods, and considerably better assessments of document coherence.
10 pages, 3 figures
References in corpus (6)
- Sequence to Sequence Learning with Neural Networks
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- On the difficulty of training Recurrent Neural Networks
- A Hierarchical Neural Autoencoder for Paragraphs and Documents
- Modelling, Visualising and Summarising Documents with a Single Convolutional Neural Network
- Larger-Context Language Modelling
Cited by in corpus (19)
- Exploring the Limits of Language Modeling
- Contextual LSTM (CLSTM) models for Large scale NLP tasks
- Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
- Representation Learning for Natural Language Processing
- LSTM based Conversation Models
- Natural Language Understanding with Distributed Representation
- Context Gates for Neural Machine Translation
- Calibration, Entropy Rates, and Memory in Language Models
- Learning to Remember Translation History with a Continuous Cache
- Reference-Aware Language Models
- Cross-Attention End-to-End ASR for Two-Party Conversations
- Context Models for OOV Word Translation in Low-Resource Languages
- Improving Context Aware Language Models
- Low-Rank RNN Adaptation for Context-Aware Language Modeling
- Dialog-context aware end-to-end speech recognition
- Do Long-Range Language Models Actually Use Long-Range Context?
- Dialog Context Language Modeling with Recurrent Neural Networks
- Acoustic-to-Word Models with Conversational Context Information
- SAM: Semantic Attribute Modulation for Language Modeling and Style Variation