Low-Rank RNN Adaptation for Context-Aware Language Modeling
arXiv:1710.02603
Abstract
A context-aware language model uses location, user and/or domain metadata (context) to adapt its predictions. In neural language models, context information is typically represented as an embedding and it is given to the RNN as an additional input, which has been shown to be useful in many applications. We introduce a more powerful mechanism for using context to adapt an RNN by letting the context vector control a low-rank transformation of the recurrent layer weight matrix. Experiments show that allowing a greater fraction of the model parameters to be adjusted has benefits in terms of perplexity and classification for several different types of context.
Accepted to TACL
References in corpus (7)
- Learning to Generate Reviews and Discovering Sentiment
- Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
- TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency
- Generative and Discriminative Text Classification with Recurrent Neural Networks
- Factorization tricks for LSTM networks
- Context-aware Natural Language Generation with Recurrent Neural Networks
- On Using Very Large Target Vocabulary for Neural Machine Translation