Sparse Activity and Sparse Connectivity in Supervised Learning

Abstract

Sparseness is a useful regularizer for learning in a wide range of applications, in particular in neural networks. This paper proposes a model targeted at classification tasks, where sparse activity and sparse connectivity are used to enhance classification capabilities. The tool for achieving this is a sparseness-enforcing projection operator which finds the closest vector with a pre-defined sparseness for any given vector. In the theoretical part of this paper, a comprehensive theory for such a projection is developed. In conclusion, it is shown that the projection is differentiable almost everywhere and can thus be implemented as a smooth neuronal transfer function. The entire model can hence be tuned end-to-end using gradient-based methods. Experiments on the MNIST database of handwritten digits show that classification performance can be boosted by sparse activity or sparse connectivity. With a combination of both, performance can be significantly better compared to classical non-sparse approaches.

See http://jmlr.org/papers/v14/thom13a.html for the authoritative version

References in corpus (3)

• Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition
• Supervised Dictionary Learning
• A note on the group lasso and a sparse group lasso

Cited by in corpus (6)

• Learning Halfspaces and Neural Networks with Random Initialization
• Efficient Dictionary Learning with Sparseness-Enforcing Projections
• Sparseout: Controlling Sparsity in Deep Networks
• Stochastic In-Face Frank-Wolfe Methods for Non-Convex Optimization and Sparse Neural Network Training
• Shakeout: A New Approach to Regularized Deep Neural Network Training
• Sparse Multi-Family Deep Scattering Network