Hierarchical Indian Buffet Neural Networks for Bayesian Continual Learning

Abstract

We place an Indian Buffet process (IBP) prior over the structure of a Bayesian Neural Network (BNN), thus allowing the complexity of the BNN to increase and decrease automatically. We further extend this model such that the prior on the structure of each hidden layer is shared globally across all layers, using a Hierarchical-IBP (H-IBP). We apply this model to the problem of resource allocation in Continual Learning (CL) where new tasks occur and the network requires extra resources. Our model uses online variational inference with reparameterisation of the Bernoulli and Beta distributions, which constitute the IBP and H-IBP priors. As we automatically learn the number of weights in each layer of the BNN, overfitting and underfitting problems are largely overcome. We show empirically that our approach offers a competitive edge over existing methods in CL.

arXiv comment22 pages, 19 figures including references and appendix. Accepted at UAI 2021

References in corpus (23)

• Auto-Encoding Variational Bayes
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Continual Learning Through Synaptic Intelligence
• An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
• Three scenarios for continual learning
• Variational Dropout and the Local Reparameterization Trick
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Reinforced Continual Learning
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• Implicit Reparameterization Gradients
• Practical Deep Learning with Bayesian Principles
• A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning
• Continual Unsupervised Representation Learning
• Monte Carlo Gradient Estimation in Machine Learning
• Task Agnostic Continual Learning Using Online Variational Bayes
• Pathwise Derivatives Beyond the Reparameterization Trick
• Continual Deep Learning by Functional Regularisation of Memorable Past
• Structured Stochastic Variational Inference
• A Unifying Bayesian View of Continual Learning
• A Slice Sampler for Restricted Hierarchical Beta Process with Applications to Shared Subspace Learning
• Bayesian Structure Adaptation for Continual Learning

Cited by in corpus (6)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Continual Learning using a Bayesian Nonparametric Dictionary of Weight Factors
• On Sequential Bayesian Inference for Continual Learning
• DrNAS: Dirichlet Neural Architecture Search
• Variational Auto-Regressive Gaussian Processes for Continual Learning
• Posterior distributions for Hierarchical Spike and Slab Indian Buffet processes