Gaussian Gated Linear Networks

Abstract

We propose the Gaussian Gated Linear Network (G-GLN), an extension to the recently proposed GLN family of deep neural networks. Instead of using backpropagation to learn features, GLNs have a distributed and local credit assignment mechanism based on optimizing a convex objective. This gives rise to many desirable properties including universality, data-efficient online learning, trivial interpretability and robustness to catastrophic forgetting. We extend the GLN framework from classification to multiple regression and density modelling by generalizing geometric mixing to a product of Gaussian densities. The G-GLN achieves competitive or state-of-the-art performance on several univariate and multivariate regression benchmarks, and we demonstrate its applicability to practical tasks including online contextual bandits and density estimation via denoising.

References in corpus (12)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Semi-Supervised Learning with Ladder Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Training Neural Networks with Local Error Signals
• Generalized Product of Experts for Automatic and Principled Fusion of Gaussian Process Predictions
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• Superposition of many models into one
• Online Learning with Gated Linear Networks
• Online Learning in Contextual Bandits using Gated Linear Networks
• Product Kanerva Machines: Factorized Bayesian Memory