TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency

Abstract

In this paper, we propose TopicRNN, a recurrent neural network (RNN)-based language model designed to directly capture the global semantic meaning relating words in a document via latent topics. Because of their sequential nature, RNNs are good at capturing the local structure of a word sequence - both semantic and syntactic - but might face difficulty remembering long-range dependencies. Intuitively, these long-range dependencies are of semantic nature. In contrast, latent topic models are able to capture the global underlying semantic structure of a document but do not account for word ordering. The proposed TopicRNN model integrates the merits of RNNs and latent topic models: it captures local (syntactic) dependencies using an RNN and global (semantic) dependencies using latent topics. Unlike previous work on contextual RNN language modeling, our model is learned end-to-end. Empirical results on word prediction show that TopicRNN outperforms existing contextual RNN baselines. In addition, TopicRNN can be used as an unsupervised feature extractor for documents. We do this for sentiment analysis on the IMDB movie review dataset and report an error rate of $6.28\%$. This is comparable to the state-of-the-art $5.91\%$ resulting from a semi-supervised approach. Finally, TopicRNN also yields sensible topics, making it a useful alternative to document models such as latent Dirichlet allocation.

International Conference on Learning Representations

Cited by in corpus (33)

• Deep Learning Based Text Classification: A Comprehensive Review
• Learning to Generate Reviews and Discovering Sentiment
• Discovering Discrete Latent Topics with Neural Variational Inference
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• A Survey on Text Classification: From Shallow to Deep Learning
• TDAM: a Topic-Dependent Attention Model for Sentiment Analysis
• Deep Learning-based Sentiment Classification: A Comparative Survey
• Generative Neural Machine Translation
• Adversarially Regularized Autoencoders
• Avoiding Latent Variable Collapse With Generative Skip Models
• Semi-Amortized Variational Autoencoders
• A Tutorial on Deep Latent Variable Models of Natural Language
• A Discrete Variational Recurrent Topic Model without the Reparametrization Trick
• Topic Memory Networks for Short Text Classification
• On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis
• Latent Topic Conversational Models
• Low-Rank RNN Adaptation for Context-Aware Language Modeling
• Goal-driven Command Recommendations for Analysts
• Behavior Gated Language Models
• Reverse Transfer Learning: Can Word Embeddings Trained for Different NLP Tasks Improve Neural Language Models?
• Non-local Recurrent Neural Memory for Supervised Sequence Modeling
• Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer
• Hierarchical Context enabled Recurrent Neural Network for Recommendation
• Stuck? No worries!: Task-aware Command Recommendation and Proactive Help for Analysts
• Learning low dimensional word based linear classifiers using Data Shared Adaptive Bootstrap Aggregated Lasso with application to IMDb data
• Harmonic-Percussive Source Separation with Deep Neural Networks and Phase Recovery
• Improving latent variable descriptiveness with AutoGen
• Explaining Away Syntactic Structure in Semantic Document Representations
• Microblog Topic Identification using Linked Open Data
• Variational Hetero-Encoder Randomized GANs for Joint Image-Text Modeling
• Recurrent Coupled Topic Modeling over Sequential Documents
• Towards Controlled Transformation of Sentiment in Sentences
• SECTOR: A Neural Model for Coherent Topic Segmentation and Classification