Icebreaker: Element-wise Active Information Acquisition with Bayesian Deep Latent Gaussian Model

Abstract

In this paper we introduce the ice-start problem, i.e., the challenge of deploying machine learning models when only little or no training data is initially available, and acquiring each feature element of data is associated with costs. This setting is representative for the real-world machine learning applications. For instance, in the health-care domain, when training an AI system for predicting patient metrics from lab tests, obtaining every single measurement comes with a high cost. Active learning, where only the label is associated with a cost does not apply to such problem, because performing all possible lab tests to acquire a new training datum would be costly, as well as unnecessary due to redundancy. We propose Icebreaker, a principled framework to approach the ice-start problem. Icebreaker uses a full Bayesian Deep Latent Gaussian Model (BELGAM) with a novel inference method. Our proposed method combines recent advances in amortized inference and stochastic gradient MCMC to enable fast and accurate posterior inference. By utilizing BELGAM's ability to fully quantify model uncertainty, we also propose two information acquisition functions for imputation and active prediction problems. We demonstrate that BELGAM performs significantly better than the previous VAE (Variational autoencoder) based models, when the data set size is small, using both machine learning benchmarks and real-world recommender systems and health-care applications. Moreover, based on BELGAM, Icebreaker further improves the performance and demonstrate the ability to use minimum amount of the training data to obtain the highest test time performance.

References in corpus (9)

• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Bayesian Active Learning for Classification and Preference Learning
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• Generative Adversarial Active Learning
• Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization
• Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo
• EDDI: Efficient Dynamic Discovery of High-Value Information with Partial VAE
• Active Feature Acquisition with Supervised Matrix Completion

Cited by in corpus (6)

• Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection
• VAEM: a Deep Generative Model for Heterogeneous Mixed Type Data
• VarFA: A Variational Factor Analysis Framework For Efficient Bayesian Learning Analytics
• Addressing practical challenges in Active Learning via a hybrid query strategy
• CLOPS: Continual Learning of Physiological Signals
• SoQal: Selective Oracle Questioning in Active Learning