Deep Learning: A Bayesian Perspective

Abstract

Deep learning is a form of machine learning for nonlinear high dimensional pattern matching and prediction. By taking a Bayesian probabilistic perspective, we provide a number of insights into more efficient algorithms for optimisation and hyper-parameter tuning. Traditional high-dimensional data reduction techniques, such as principal component analysis (PCA), partial least squares (PLS), reduced rank regression (RRR), projection pursuit regression (PPR) are all shown to be shallow learners. Their deep learning counterparts exploit multiple deep layers of data reduction which provide predictive performance gains. Stochastic gradient descent (SGD) training optimisation and Dropout (DO) regularization provide estimation and variable selection. Bayesian regularization is central to finding weights and connections in networks to optimize the predictive bias-variance trade-off. To illustrate our methodology, we provide an analysis of international bookings on Airbnb. Finally, we conclude with directions for future research.

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• XGBoost: A Scalable Tree Boosting System
• Deep Learning in Neural Networks: An Overview
• Practical Bayesian Optimization of Machine Learning Algorithms
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• BART: Bayesian additive regression trees
• Deep Learning for Short-Term Traffic Flow Prediction
• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Fisher Lecture: Dimension Reduction in Regression
• Neural Variational Inference and Learning in Belief Networks
• A Central Limit Theorem for Convex Sets
• Distributed optimization of deeply nested systems
• A Statistical Theory of Deep Learning via Proximal Splitting

Cited by in corpus (22)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons
• Weighted Bayesian Bootstrap for Scalable Bayes
• Bayesian graph convolutional neural networks via tempered MCMC
• Revisiting Bayesian Autoencoders with MCMC
• Decentralized Stochastic Gradient Langevin Dynamics and Hamiltonian Monte Carlo
• Uncertainty Quantification for Sparse Deep Learning
• Marginally-calibrated deep distributional regression
• Prior choice affects ability of Bayesian neural networks to identify unknowns
• Data Augmentation for Bayesian Deep Learning
• Deep Learning Partial Least Squares
• Predictive Learning: Using Future Representation Learning Variantial Autoencoder for Human Action Prediction
• Leveraging Uncertainty in Deep Learning for Selective Classification
• Informative Bayesian Neural Network Priors for Weak Signals
• Bayesian Neural Networks for Virtual Flow Metering: An Empirical Study
• Scalable Modeling of Spatiotemporal Data using the Variational Autoencoder: an Application in Glaucoma
• Merging Two Cultures: Deep and Statistical Learning
• Seeing Convolution Through the Eyes of Finite Transformation Semigroup Theory: An Abstract Algebraic Interpretation of Convolutional Neural Networks
• Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning
• Comparison of Deep Neural Networks and Deep Hierarchical Models for Spatio-Temporal Data
• Detecting conflicting summary statistics in likelihood-free inference