Explicitly Bayesian Regularizations in Deep Learning

Abstract

Generalization is essential for deep learning. In contrast to previous works claiming that Deep Neural Networks (DNNs) have an implicit regularization implemented by the stochastic gradient descent, we demonstrate explicitly Bayesian regularizations in a specific category of DNNs, i.e., Convolutional Neural Networks (CNNs). First, we introduce a novel probabilistic representation for the hidden layers of CNNs and demonstrate that CNNs correspond to Bayesian networks with the serial connection. Furthermore, we show that the hidden layers close to the input formulate prior distributions, thus CNNs have explicitly Bayesian regularizations based on the Bayesian regularization theory. In addition, we clarify two recently observed empirical phenomena that are inconsistent with traditional theories of generalization. Finally, we validate the proposed theory on a synthetic dataset

References in corpus (10)

• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• An exact mapping between the Variational Renormalization Group and Deep Learning
• Statistical exponential families: A digest with flash cards
• Sharp Minima Can Generalize For Deep Nets
• Deep Convolutional Networks as shallow Gaussian Processes
• Implicit Regularization in Deep Matrix Factorization
• A Probabilistic Framework for Deep Learning
• Non-Gaussian processes and neural networks at finite widths
• A synthetic dataset for deep learning