Gibbs Max-margin Topic Models with Data Augmentation

Abstract

Max-margin learning is a powerful approach to building classifiers and structured output predictors. Recent work on max-margin supervised topic models has successfully integrated it with Bayesian topic models to discover discriminative latent semantic structures and make accurate predictions for unseen testing data. However, the resulting learning problems are usually hard to solve because of the non-smoothness of the margin loss. Existing approaches to building max-margin supervised topic models rely on an iterative procedure to solve multiple latent SVM subproblems with additional mean-field assumptions on the desired posterior distributions. This paper presents an alternative approach by defining a new max-margin loss. Namely, we present Gibbs max-margin supervised topic models, a latent variable Gibbs classifier to discover hidden topic representations for various tasks, including classification, regression and multi-task learning. Gibbs max-margin supervised topic models minimize an expected margin loss, which is an upper bound of the existing margin loss derived from an expected prediction rule. By introducing augmented variables and integrating out the Dirichlet variables analytically by conjugacy, we develop simple Gibbs sampling algorithms with no restricting assumptions and no need to solve SVM subproblems. Furthermore, each step of the "augment-and-collapse" Gibbs sampling algorithms has an analytical conditional distribution, from which samples can be easily drawn. Experimental results demonstrate significant improvements on time efficiency. The classification performance is also significantly improved over competitors on binary, multi-class and multi-label classification tasks.

35 pages

References in corpus (2)

• Supervised Topic Models
• Max-Margin Nonparametric Latent Feature Models for Link Prediction

Cited by in corpus (14)

• Max-Margin Nonparametric Latent Feature Models for Link Prediction
• Online Bayesian Passive-Aggressive Learning
• Boosting Deep Learning Risk Prediction with Generative Adversarial Networks for Electronic Health Records
• Max-margin Deep Generative Models
• End-to-end Learning of LDA by Mirror-Descent Back Propagation over a Deep Architecture
• Dropout Training for Support Vector Machines
• The Homophily Principle in Social Network Analysis
• Diversity-Promoting Bayesian Learning of Latent Variable Models
• Discriminative Nonparametric Latent Feature Relational Models with Data Augmentation
• Deep Autoencoding Topic Model with Scalable Hybrid Bayesian Inference
• Fast Sampling for Bayesian Max-Margin Models
• Dropout Training for SVMs with Data Augmentation
• Spectral Learning for Supervised Topic Models
• Learning beyond Predefined Label Space via Bayesian Nonparametric Topic Modelling