Variational Bayesian Unlearning

Abstract

This paper studies the problem of approximately unlearning a Bayesian model from a small subset of the training data to be erased. We frame this problem as one of minimizing the Kullback-Leibler divergence between the approximate posterior belief of model parameters after directly unlearning from erased data vs. the exact posterior belief from retraining with remaining data. Using the variational inference (VI) framework, we show that it is equivalent to minimizing an evidence upper bound which trades off between fully unlearning from erased data vs. not entirely forgetting the posterior belief given the full data (i.e., including the remaining data); the latter prevents catastrophic unlearning that can render the model useless. In model training with VI, only an approximate (instead of exact) posterior belief given the full data can be obtained, which makes unlearning even more challenging. We propose two novel tricks to tackle this challenge. We empirically demonstrate our unlearning methods on Bayesian models such as sparse Gaussian process and logistic regression using synthetic and real-world datasets.

34th Annual Conference on Neural Information Processing Systems (NeurIPS 2020), Extended version with proofs, 22 pages

References in corpus (7)

• Variational Inference with Normalizing Flows
• Gaussian Processes for Big Data
• Decentralized Data Fusion and Active Sensing with Mobile Sensors for Modeling and Predicting Spatiotemporal Traffic Phenomena
• Parallel Gaussian Process Regression with Low-Rank Covariance Matrix Approximations
• Certified Data Removal from Machine Learning Models
• Implicit Posterior Variational Inference for Deep Gaussian Processes
• Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models - a Gentle Tutorial

Cited by in corpus (6)

• Zero-Shot Machine Unlearning
• Markov Chain Monte Carlo-Based Machine Unlearning: Unlearning What Needs to be Forgotten
• Coded Machine Unlearning
• ERASER: Machine Unlearning in MLaaS via an Inference Serving-Aware Approach
• Feature Unlearning for Pre-trained GANs and VAEs
• Bayesian Variational Federated Learning and Unlearning in Decentralized Networks