Behavior Gated Language Models
arXiv:1909.00107
Abstract
Most current language modeling techniques only exploit co-occurrence, semantic and syntactic information from the sequence of words. However, a range of information such as the state of the speaker and dynamics of the interaction might be useful. In this work we derive motivation from psycholinguistics and propose the addition of behavioral information into the context of language modeling. We propose the augmentation of language models with an additional module which analyzes the behavioral state of the current context. This behavioral information is used to gate the outputs of the language model before the final word prediction output. We show that the addition of behavioral context in language models achieves lower perplexities on behavior-rich datasets. We also confirm the validity of the proposed models on a variety of model architectures and improve on previous state-of-the-art models with generic domain Penn Treebank Corpus.
References in corpus (7)
- Recurrent Neural Network Regularization
- Contextual LSTM (CLSTM) models for Large scale NLP tasks
- TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency
- Trellis Networks for Sequence Modeling
- Topic Compositional Neural Language Model
- Dynamic Evaluation of Transformer Language Models
- Partially Shuffling the Training Data to Improve Language Models