Winner-Take-All Autoencoders
arXiv:1409.2752
Abstract
In this paper, we propose a winner-take-all method for learning hierarchical sparse representations in an unsupervised fashion. We first introduce fully-connected winner-take-all autoencoders which use mini-batch statistics to directly enforce a lifetime sparsity in the activations of the hidden units. We then propose the convolutional winner-take-all autoencoder which combines the benefits of convolutional architectures and autoencoders for learning shift-invariant sparse representations. We describe a way to train convolutional autoencoders layer by layer, where in addition to lifetime sparsity, a spatial sparsity within each feature map is achieved using winner-take-all activation functions. We will show that winner-take-all autoencoders can be used to to learn deep sparse representations from the MNIST, CIFAR-10, ImageNet, Street View House Numbers and Toronto Face datasets, and achieve competitive classification performance.
References in corpus (4)
Cited by in corpus (13)
- How Can We Be So Dense? The Benefits of Using Highly Sparse Representations
- ACCNet: Actor-Coordinator-Critic Net for "Learning-to-Communicate" with Deep Multi-agent Reinforcement Learning
- KATE: K-Competitive Autoencoder for Text
- Anomaly Detection with Tensor Networks
- A Probabilistic Framework for Deep Learning
- Learning Multi-agent Communication under Limited-bandwidth Restriction for Internet Packet Routing
- Learning Agent Communication under Limited Bandwidth by Message Pruning
- Information Potential Auto-Encoders
- Semi-Supervised Learning with the Deep Rendering Mixture Model
- Deep Neural Network Assisted Iterative Reconstruction Method for Low Dose CT
- Exploiting Spatio-Temporal Structure with Recurrent Winner-Take-All Networks
- Location-aware Upsampling for Semantic Segmentation
- Sparse associative memory based on contextual code learning for disambiguating word senses