Bayesian Sparsification Methods for Deep Complex-valued Networks

Abstract

With continual miniaturization ever more applications of deep learning can be found in embedded systems, where it is common to encounter data with natural complex domain representation. To this end we extend Sparse Variational Dropout to complex-valued neural networks and verify the proposed Bayesian technique by conducting a large numerical study of the performance-compression trade-off of C-valued networks on two tasks: image recognition on MNIST-like and CIFAR10 datasets and music transcription on MusicNet. We replicate the state-of-the-art result by Trabelsi et al. [2018] on MusicNet with a complex-valued network compressed by 50-100x at a small performance penalty.

arXiv commentFindings and conclusions unchanged. Improved overall presentation and redid the plots with larger markers and annotations. Coherent story about compression, CVNN, BI to SGVB with local reparameterization and additive noise. Better coverage in lit-review, clearer connections of Dropout to Bayes, VD, and pruning

References in corpus (29)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Auto-Encoding Variational Bayes
• Distilling the Knowledge in a Neural Network
• Improving neural networks by preventing co-adaptation of feature detectors
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Categorical Reparameterization with Gumbel-Softmax
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Tensorizing Neural Networks
• EMNIST: an extension of MNIST to handwritten letters
• Variational Dropout and the Local Reparameterization Trick
• The State of Sparsity in Deep Neural Networks
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Training deep neural networks with low precision multiplications
• Learning Features of Music from Scratch
• Implicit Reparameterization Gradients
• Full-Capacity Unitary Recurrent Neural Networks
• Associative Long Short-Term Memory
• On Complex Valued Convolutional Neural Networks
• Operator Variational Inference
• Mixed Precision DNNs: All you need is a good parametrization
• Superposition of many models into one
• Complex Gated Recurrent Neural Networks
• On the Computation of Complex-valued Gradients with Application to Statistically Optimum Beamforming
• Evaluation of Complex-Valued Neural Networks on Real-Valued Classification Tasks
• Variational Dropout via Empirical Bayes