A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model

Abstract

Inspired by the success of Convolutional Neural Networks (CNNs) for supervised prediction in images, we design the Deconvolutional Generative Model (DGM), a new probabilistic generative model whose inference calculations correspond to those in a given CNN architecture. The DGM uses a CNN to design the prior distribution in the probabilistic model. Furthermore, the DGM generates images from coarse to finer scales. It introduces a small set of latent variables at each scale, and enforces dependencies among all the latent variables via a conjugate prior distribution. This conjugate prior yields a new regularizer based on paths rendered in the generative model for training CNNs-the Rendering Path Normalization (RPN). We demonstrate that this regularizer improves generalization, both in theory and in practice. In addition, likelihood estimation in the DGM yields training losses for CNNs, and inspired by this, we design a new loss termed as the Max-Min cross entropy which outperforms the traditional cross-entropy loss for object classification. The Max-Min cross entropy suggests a new deep network architecture, namely the Max-Min network, which can learn from less labeled data while maintaining good prediction performance. Our experiments demonstrate that the DGM with the RPN and the Max-Min architecture exceeds or matches the-state-of-art on benchmarks including SVHN, CIFAR10, and CIFAR100 for semi-supervised and supervised learning tasks.

arXiv commentKeywords: neural nets, generative models, semi-supervised learning, cross-entropy, statistical guarantees 80 pages, 7 figures, 8 tables

References in corpus (18)

• Auto-Encoding Variational Bayes
• Improved Regularization of Convolutional Neural Networks with Cutout
• The information bottleneck method
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Semi-Supervised Learning with Deep Generative Models
• NICE: Non-linear Independent Components Estimation
• The Loss Surfaces of Multilayer Networks
• Understanding Deep Convolutional Networks
• Semi-Supervised Learning with Ladder Networks
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• Working Locally Thinking Globally: Theoretical Guarantees for Convolutional Sparse Coding
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Semi-supervised Learning with GANs: Manifold Invariance with Improved Inference
• Mathematics of Deep Learning
• A Probabilistic Theory of Deep Learning
• Topology and Geometry of Half-Rectified Network Optimization
• A Probabilistic Framework for Deep Learning
• Semi-Supervised Learning with the Deep Rendering Mixture Model

Cited by in corpus (4)

• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• Neural Networks with Recurrent Generative Feedback
• Manifold Graph with Learned Prototypes for Semi-Supervised Image Classification
• Bio-inspired Robustness: A Review