Sparsely Activated Networks: A new method for decomposing and compressing data

Abstract

Recent literature on unsupervised learning focused on designing structural priors with the aim of learning meaningful features, but without considering the description length of the representations. In this thesis, first we introduce the $\phi$ metric that evaluates unsupervised models based on their reconstruction accuracy and the degree of compression of their internal representations. We then present and define two activation functions (Identity, ReLU) as base of reference and three sparse activation functions (top-k absolutes, Extrema-Pool indices, Extrema) as candidate structures that minimize the previously defined metric $\phi$. We lastly present Sparsely Activated Networks (SANs) that consist of kernels with shared weights that, during encoding, are convolved with the input and then passed through a sparse activation function. During decoding, the same weights are convolved with the sparse activation map and subsequently the partial reconstructions from each weight are summed to reconstruct the input. We compare SANs using the five previously defined activation functions on a variety of datasets (Physionet, UCI-epilepsy, MNIST, FMNIST) and show that models that are selected using $\phi$ have small description representation length and consist of interpretable kernels.

arXiv commentPhD Thesis in Greek, 158 pages for the main text, 23 supplementary pages for the presentation. arXiv:1907.06592, arXiv:1904.13216, arXiv:1902.11122

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Survey on Deep Learning in Medical Image Analysis
• Auto-Encoding Variational Bayes
• Neural Machine Translation by Jointly Learning to Align and Translate
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Deep learning with convolutional neural networks for EEG decoding and visualization
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Deeply-Supervised Nets
• Anatomically Constrained Neural Networks (ACNN): Application to Cardiac Image Enhancement and Segmentation
• Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields
• Focal onset seizure prediction using convolutional networks
• A Fully Convolutional Neural Network for Cardiac Segmentation in Short-Axis MRI
• Towards Biologically Plausible Deep Learning
• Automatic calcium scoring in low-dose chest CT using deep neural networks with dilated convolutions
• Deep learning analysis of the myocardium in coronary CT angiography for identification of patients with functionally significant coronary artery stenosis
• Improved Microaneurysm Detection using Deep Neural Networks
• Dilated Convolutional Neural Networks for Cardiovascular MR Segmentation in Congenital Heart Disease
• Detection of Inferior Myocardial Infarction using Shallow Convolutional Neural Networks
• Estimation of the volume of the left ventricle from MRI images using deep neural networks
• Highrisk Prediction from Electronic Medical Records via Deep Attention Networks
• A Labeling-Free Approach to Supervising Deep Neural Networks for Retinal Blood Vessel Segmentation
• Sparsely Activated Networks