Constructing Deep Neural Networks by Bayesian Network Structure Learning

Abstract

We introduce a principled approach for unsupervised structure learning of deep neural networks. We propose a new interpretation for depth and inter-layer connectivity where conditional independencies in the input distribution are encoded hierarchically in the network structure. Thus, the depth of the network is determined inherently. The proposed method casts the problem of neural network structure learning as a problem of Bayesian network structure learning. Then, instead of directly learning the discriminative structure, it learns a generative graph, constructs its stochastic inverse, and then constructs a discriminative graph. We prove that conditional-dependency relations among the latent variables in the generative graph are preserved in the class-conditional discriminative graph. We demonstrate on image classification benchmarks that the deepest layers (convolutional and dense) of common networks can be replaced by significantly smaller learned structures, while maintaining classification accuracy---state-of-the-art on tested benchmarks. Our structure learning algorithm requires a small computational cost and runs efficiently on a standard desktop CPU.

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Natural Language Processing (almost) from Scratch
• Neural Architecture Search with Reinforcement Learning
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Learning Transferable Features with Deep Adaptation Networks
• Densely Connected Convolutional Networks
• Wide Residual Networks
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• MADE: Masked Autoencoder for Distribution Estimation
• Net2Net: Accelerating Learning via Knowledge Transfer
• Batch-normalized Maxout Network in Network
• Automated Pruning for Deep Neural Network Compression
• Learning the Structure of Deep Sparse Graphical Models
• Inference Networks for Sequential Monte Carlo in Graphical Models
• Learning to Prune Filters in Convolutional Neural Networks
• Neural networks with differentiable structure

Cited by in corpus (3)

• Mind Mappings: Enabling Efficient Algorithm-Accelerator Mapping Space Search
• DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs
• Modeling Uncertainty by Learning a Hierarchy of Deep Neural Connections