Variational Approximation Error in Bayesian Non-negative Matrix Factorization

Abstract

Non-negative matrix factorization (NMF) is a knowledge discovery method that is used in many fields. Variational inference and Gibbs sampling methods for it are also wellknown. However, the variational approximation error has not been clarified yet, because NMF is not statistically regular and the prior distribution used in variational Bayesian NMF (VBNMF) has zero or divergence points. In this paper, using algebraic geometrical methods, we theoretically analyze the difference in negative log evidence (a.k.a. free energy) between VBNMF and Bayesian NMF, i.e., the Kullback-Leibler divergence between the variational posterior and the true posterior. We derive an upper bound for the learning coefficient (a.k.a. the real log canonical threshold) in Bayesian NMF. By using the upper bound, we find a lower bound for the approximation error, asymptotically. The result quantitatively shows how well the VBNMF algorithm can approximate Bayesian NMF; the lower bound depends on the hyperparameters and the true nonnegative rank. A numerical experiment demonstrates the theoretical result.

arXiv comment21 pages. 1 table. Revision in Neural Networks

References in corpus (4)

• Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory
• Asymptotic Model Selection for Naive Bayesian Networks
• Upper Bound of Bayesian Generalization Error in Non-negative Matrix Factorization
• Asymptotic Bayesian Generalization Error in Latent Dirichlet Allocation and Stochastic Matrix Factorization

Cited by in corpus (2)

• The Exact Asymptotic Form of Bayesian Generalization Error in Latent Dirichlet Allocation
• An Asymptotic Equation Linking WAIC and WBIC in Singular Models