Low-Rank RNN Adaptation for Context-Aware Language Modeling
arXiv:1710.02603
Abstract
A context-aware language model uses location, user and/or domain metadata (context) to adapt its predictions. In neural language models, context information is typically represented as an embedding and it is given to the RNN as an additional input, which has been shown to be useful in many applications. We introduce a more powerful mechanism for using context to adapt an RNN by letting the context vector control a low-rank transformation of the recurrent layer weight matrix. Experiments show that allowing a greater fraction of the model parameters to be adjusted has benefits in terms of perplexity and classification for several different types of context.
arXiv commentAccepted to TACL
References in corpus (14)
- Visualizing and Understanding Recurrent Networks
- Learning to Generate Reviews and Discovering Sentiment
- Contextual LSTM (CLSTM) models for Large scale NLP tasks
- A Persona-Based Neural Conversation Model
- Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
- TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency
- Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
- Generative and Discriminative Text Classification with Recurrent Neural Networks
- Factorization tricks for LSTM networks
- Recurrent Dropout without Memory Loss
- Context-aware Natural Language Generation with Recurrent Neural Networks
- Document Context Language Models
- LSTM based Conversation Models
- On Using Very Large Target Vocabulary for Neural Machine Translation