Towards Big Topic Modeling
arXiv:1311.4150
Abstract
To solve the big topic modeling problem, we need to reduce both time and space complexities of batch latent Dirichlet allocation (LDA) algorithms. Although parallel LDA algorithms on the multi-processor architecture have low time and space complexities, their communication costs among processors often scale linearly with the vocabulary size and the number of topics, leading to a serious scalability problem. To reduce the communication complexity among processors for a better scalability, we propose a novel communication-efficient parallel topic modeling architecture based on power law, which consumes orders of magnitude less communication time when the number of topics is large. We combine the proposed communication-efficient parallel architecture with the online belief propagation (OBP) algorithm referred to as POBP for big topic modeling tasks. Extensive empirical results confirm that POBP has the following advantages to solve the big topic modeling problem: 1) high accuracy, 2) communication-efficient, 3) fast speed, and 4) constant memory usage when compared with recent state-of-the-art parallel LDA algorithms on the multi-processor architecture.
arXiv comment14 pages
References in corpus (8)
- Online EM Algorithm for Latent Data Models
- On Smoothing and Inference for Topic Models
- Residual Belief Propagation: Informed Scheduling for Asynchronous Message Passing
- Sparse Stochastic Inference for Latent Dirichlet allocation
- Learning Topic Models by Belief Propagation
- A Topic Modeling Toolbox Using Belief Propagation
- A New Approach to Speeding Up Topic Modeling
- Communication-Efficient Parallel Belief Propagation for Latent Dirichlet Allocation