DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM

Abstract

In the burgeoning field of natural language processing (NLP), Neural Topic Models (NTMs) , Large Language Models (LLMs) and Diffusion model have emerged as areas of significant research interest. Despite this, NTMs primarily utilize contextual embeddings from LLMs, which are not optimal for clustering or capable for topic based text generation. NTMs have never been combined with diffusion model for text generation. Our study addresses these gaps by introducing a novel framework named Diffusion-Enhanced Topic Modeling using Encoder-Decoder-based LLMs (DeTiME). DeTiME leverages Encoder-Decoder-based LLMs to produce highly clusterable embeddings that could generate topics that exhibit both superior clusterability and enhanced semantic coherence compared to existing methods. Additionally, by exploiting the power of diffusion model, our framework also provides the capability to do topic based text generation. This dual functionality allows users to efficiently produce highly clustered topics and topic based text generation simultaneously. DeTiME's potential extends to generating clustered embeddings as well. Notably, our proposed framework(both encoder-decoder based LLM and diffusion model) proves to be efficient to train and exhibits high adaptability to other LLMs and diffusion model, demonstrating its potential for a wide array of applications.

19 pages, 4 figures, EMNLP 2023

References in corpus (15)

• Generative Adversarial Networks
• BERTopic: Neural topic modeling with a class-based TF-IDF procedure
• Scaling Instruction-Finetuned Language Models
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
• The Creativity of Text-to-Image Generation
• Diffusion-LM Improves Controllable Text Generation
• The Curse of Recursion: Training on Generated Data Makes Models Forget
• DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models
• Topic-Guided Variational Autoencoders for Text Generation
• Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise
• Diffusion Models in NLP: A Survey
• Empowering Diffusion Models on the Embedding Space for Text Generation
• Effective Neural Topic Modeling with Embedding Clustering Regularization
• vONTSS: vMF based semi-supervised neural topic modeling with optimal transport
• KDSTM: Neural Semi-supervised Topic Modeling with Knowledge Distillation

Cited by in corpus (2)

• Topic Classification of Case Law Using a Large Language Model and a New Taxonomy for UK Law: AI Insights into Summary Judgment
• LentEx: Generalizable Latent Entity Extraction via Synthetic Data and Instruction-Tuned LLMs