Understanding Locally Competitive Networks
arXiv:1410.1165
Abstract
Recently proposed neural network activation functions such as rectified linear, maxout, and local winner-take-all have allowed for faster and more effective training of deep neural architectures on large and complex datasets. The common trait among these functions is that they implement local competition between small groups of computational units within a layer, so that only part of the network is activated for any given input pattern. In this paper, we attempt to visualize and understand this self-modularization, and suggest a unified explanation for the beneficial properties of such networks. We also show how our insights can be directly useful for efficiently performing retrieval over large datasets using neural networks.
9 pages + 2 supplementary, Accepted to ICLR 2015 Conference track
References in corpus (5)
- Improving neural networks by preventing co-adaptation of feature detectors
- DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
- Pylearn2: a machine learning research library
- On the number of response regions of deep feed forward networks with piece-wise linear activations
- Sparse similarity-preserving hashing
Cited by in corpus (6)
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Deep Convolutional Neural Network Design Patterns
- Enhancing Adversarial Defense by k-Winners-Take-All
- Deep Competitive Pathway Networks
- Eliminating Catastrophic Interference with Biased Competition
- Neural Path Features and Neural Path Kernel : Understanding the role of gates in deep learning