Efficient Online Bayesian Inference for Neural Bandits

Abstract

In this paper we present a new algorithm for online (sequential) inference in Bayesian neural networks, and show its suitability for tackling contextual bandit problems. The key idea is to combine the extended Kalman filter (which locally linearizes the likelihood function at each time step) with a (learned or random) low-dimensional affine subspace for the parameters; the use of a subspace enables us to scale our algorithm to models with $\sim 1M$ parameters. While most other neural bandit methods need to store the entire past dataset in order to avoid the problem of "catastrophic forgetting", our approach uses constant memory. This is possible because we represent uncertainty about all the parameters in the model, not just the final linear layer. We show good results on the "Deep Bayesian Bandit Showdown" benchmark, as well as MNIST and a recommender system.

References in corpus (13)

• Overcoming catastrophic forgetting in neural networks
• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Thompson Sampling for Contextual Bandits with Linear Payoffs
• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
• When Do Neural Networks Outperform Kernel Methods?
• The Case for Bayesian Deep Learning
• Neural Contextual Bandits with UCB-based Exploration
• Laplace Redux -- Effortless Bayesian Deep Learning
• On Multi-Armed Bandit Designs for Dose-Finding Clinical Trials
• Thompson Sampling with Approximate Inference
• How many degrees of freedom do we need to train deep networks: a loss landscape perspective