Variational Relevance Vector Machines

Abstract

The Support Vector Machine (SVM) of Vapnik (1998) has become widely established as one of the leading approaches to pattern recognition and machine learning. It expresses predictions in terms of a linear combination of kernel functions centred on a subset of the training data, known as support vectors. Despite its widespread success, the SVM suffers from some important limitations, one of the most significant being that it makes point predictions rather than generating predictive distributions. Recently Tipping (1999) has formulated the Relevance Vector Machine (RVM), a probabilistic model whose functional form is equivalent to the SVM. It achieves comparable recognition accuracy to the SVM, yet provides a full predictive distribution, and also requires substantially fewer kernel functions. The original treatment of the RVM relied on the use of type II maximum likelihood (the `evidence framework') to provide point estimates of the hyperparameters which govern model sparsity. In this paper we show how the RVM can be formulated and solved within a completely Bayesian paradigm through the use of variational inference, thereby giving a posterior distribution over both parameters and hyperparameters. We demonstrate the practicality and performance of the variational RVM using both synthetic and real world examples.

Appears in Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence (UAI2000)

Cited by in corpus (28)

• Sparse Estimation using Bayesian Hierarchical Prior Modeling for Real and Complex Linear Models
• Lognormal and Gamma Mixed Negative Binomial Regression
• Posterior consistency in linear models under shrinkage priors
• Semi-Implicit Graph Variational Auto-Encoders
• Probabilistic Feature Selection and Classification Vector Machine
• Covariance-Free Sparse Bayesian Learning
• Predictive Coarse-Graining
• A physics-aware, probabilistic machine learning framework for coarse-graining high-dimensional systems in the Small Data regime
• Incorporating physical constraints in a deep probabilistic machine learning framework for coarse-graining dynamical systems
• Bayesian model and dimension reduction for uncertainty propagation: applications in random media
• Semi-Implicit Variational Inference
• Sparse Polynomial Chaos expansions using Variational Relevance Vector Machines
• Non-parametric Bayesian Models of Response Function in Dynamic Image Sequences
• Learning Interpretable Deep State Space Model for Probabilistic Time Series Forecasting
• Automatic Relevance Determination For Deep Generative Models
• Sparse online variational Bayesian regression
• A Sparse Bayesian Deep Learning Approach for Identification of Cascaded Tanks Benchmark
• Fully Bayesian Analysis of the Relevance Vector Machine Classification for Imbalanced Data
• Bayesian stochastic multi-scale analysis via energy considerations
• Relevance Vector Machines for classifying points and regions in biological sequences
• Hierarchical sparsity priors for regression models
• Mean field variational Bayesian inference for support vector machine classification
• Joint Channel Estimation and Data Detection for Multihop OFDM Relaying System under Unknown Channel Orders and Doppler Frequencies
• Complexity-Optimized Sparse Bayesian Learning for Scalable Classification Tasks
• Relevance Vector Machine with Weakly Informative Hyperprior and Extended Predictive Information Criterion
• Asynchronous Distributed Variational Gaussian Processes for Regression
• A data-driven model order reduction approach for Stokes flow through random porous media
• Thompson Sampling via Local Uncertainty