Diversity Networks: Neural Network Compression Using Determinantal Point Processes

Abstract

We introduce Divnet, a flexible technique for learning networks with diverse neurons. Divnet models neuronal diversity by placing a Determinantal Point Process (DPP) over neurons in a given layer. It uses this DPP to select a subset of diverse neurons and subsequently fuses the redundant neurons into the selected ones. Compared with previous approaches, Divnet offers a more principled, flexible technique for capturing neuronal diversity and thus implicitly enforcing regularization. This enables effective auto-tuning of network architecture and leads to smaller network sizes without hurting performance. Moreover, through its focus on diversity and neuron fusing, Divnet remains compatible with other procedures that seek to reduce memory footprints of networks. We present experimental results to corroborate our claims: for pruning neural networks, Divnet is seen to be notably superior to competing approaches.

This paper appeared under the shorter title Diversity Networks at ICLR 2016 (http://www.iclr.cc/doku.php?id=iclr2016:main#accepted_papers_conference_track)

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• Deep Learning with Limited Numerical Precision
• Determinantal point processes for machine learning
• Compressing Neural Networks with the Hashing Trick
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Efficient Sampling for k-Determinantal Point Processes
• Deep Fried Convnets

Cited by in corpus (24)

• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Channel Pruning for Accelerating Very Deep Neural Networks
• Recent Advances in Convolutional Neural Networks
• N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
• What is the State of Neural Network Pruning?
• Proving the Lottery Ticket Hypothesis: Pruning is All You Need
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Understanding Neural Networks and Individual Neuron Importance via Information-Ordered Cumulative Ablation
• Resource-Efficient Neural Networks for Embedded Systems
• Kronecker Determinantal Point Processes
• Depth-wise Decomposition for Accelerating Separable Convolutions in Efficient Convolutional Neural Networks
• Exploiting Channel Similarity for Accelerating Deep Convolutional Neural Networks
• Scalable Learning and MAP Inference for Nonsymmetric Determinantal Point Processes
• Scaling Up Exact Neural Network Compression by ReLU Stability
• Baseline Pruning-Based Approach to Trojan Detection in Neural Networks
• Flexible Modeling of Diversity with Strongly Log-Concave Distributions
• Learning Diverse Representations for Fast Adaptation to Distribution Shift
• Training Sparse Neural Networks using Compressed Sensing
• DPPNet: Approximating Determinantal Point Processes with Deep Networks
• Efficient and Robust Machine Learning for Real-World Systems
• Using noise to probe recurrent neural network structure and prune synapses
• Simultaneously Learning Architectures and Features of Deep Neural Networks
• Explore the Knowledge contained in Network Weights to Obtain Sparse Neural Networks
• Statistical Mechanical Analysis of Neural Network Pruning