Sparsity-Probe: Analysis tool for Deep Learning Models
arXiv:2105.06849
Abstract
We propose a probe for the analysis of deep learning architectures that is based on machine learning and approximation theoretical principles. Given a deep learning architecture and a training set, during or after training, the Sparsity Probe allows to analyze the performance of intermediate layers by quantifying the geometrical features of representations of the training set. We show how the Sparsity Probe enables measuring the contribution of adding depth to a given architecture, to detect under-performing layers, etc., all this without any auxiliary test data set.
References in corpus (13)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Bootstrap your own latent: A new approach to self-supervised Learning
- Gaussian Error Linear Units (GELUs)
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction
- Opening the Black Box of Deep Neural Networks via Information
- AutoAugment: Learning Augmentation Policies from Data
- Deep Neural Networks as Gaussian Processes
- Deep Network Approximation for Smooth Functions
- Optimal approximation of continuous functions by very deep ReLU networks
- Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning
- Scaling Down Deep Learning with MNIST-1D