Winner-Take-All Autoencoders

Abstract

In this paper, we propose a winner-take-all method for learning hierarchical sparse representations in an unsupervised fashion. We first introduce fully-connected winner-take-all autoencoders which use mini-batch statistics to directly enforce a lifetime sparsity in the activations of the hidden units. We then propose the convolutional winner-take-all autoencoder which combines the benefits of convolutional architectures and autoencoders for learning shift-invariant sparse representations. We describe a way to train convolutional autoencoders layer by layer, where in addition to lifetime sparsity, a spatial sparsity within each feature map is achieved using winner-take-all activation functions. We will show that winner-take-all autoencoders can be used to to learn deep sparse representations from the MNIST, CIFAR-10, ImageNet, Street View House Numbers and Toronto Face datasets, and achieve competitive classification performance.

References in corpus (4)

• Improving neural networks by preventing co-adaptation of feature detectors
• Semi-Supervised Learning with Deep Generative Models
• Convolutional Kernel Networks
• Differentiable Pooling for Hierarchical Feature Learning

Cited by in corpus (36)

• Image reconstruction by domain transform manifold learning
• Learning Representations of Ultrahigh-dimensional Data for Random Distance-based Outlier Detection
• Stacked What-Where Auto-encoders
• CNN-based Density Estimation and Crowd Counting: A Survey
• Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning
• How Can We Be So Dense? The Benefits of Using Highly Sparse Representations
• Sparse Unsupervised Capsules Generalize Better
• ACCNet: Actor-Coordinator-Critic Net for "Learning-to-Communicate" with Deep Multi-agent Reinforcement Learning
• Locate, Size and Count: Accurately Resolving People in Dense Crowds via Detection
• Pixel Deconvolutional Networks
• KATE: K-Competitive Autoencoder for Text
• Anomaly Detection with Tensor Networks
• A Probabilistic Framework for Deep Learning
• Sparse Representations for Object and Ego-motion Estimation in Dynamic Scenes
• Learning Multi-agent Communication under Limited-bandwidth Restriction for Internet Packet Routing
• Top-Down Learning for Structured Labeling with Convolutional Pseudoprior
• Learning Agent Communication under Limited Bandwidth by Message Pruning
• Towards a Near Universal Time Series Data Mining Tool: Introducing the Matrix Profile
• AHA! an 'Artificial Hippocampal Algorithm' for Episodic Machine Learning
• The Utility of Sparse Representations for Control in Reinforcement Learning
• Information Potential Auto-Encoders
• Supervised COSMOS Autoencoder: Learning Beyond the Euclidean Loss!
• Semi-Supervised Learning with the Deep Rendering Mixture Model
• Deep Neural Network Assisted Iterative Reconstruction Method for Low Dose CT
• Spurious samples in deep generative models: bug or feature?
• Digits that are not: Generating new types through deep neural nets
• Exploiting Spatio-Temporal Structure with Recurrent Winner-Take-All Networks
• Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations Online
• Location-aware Upsampling for Semantic Segmentation
• Fault-Diagnosing SLAM for Varying Scale Change Detection
• Deep learning scheme for recovery of broadband microwave photonic receiving systems in transceivers without expert knowledge and system priors
• Representation Learning by Reconstructing Neighborhoods
• Sparse associative memory based on contextual code learning for disambiguating word senses
• Inducing Hierarchical Compositional Model by Sparsifying Generator Network
• Joint Regularization on Activations and Weights for Efficient Neural Network Pruning
• Unsupervised Foveal Vision Neural Networks with Top-Down Attention