k-Sparse Autoencoders

Abstract

Recently, it has been observed that when representations are learnt in a way that encourages sparsity, improved performance is obtained on classification tasks. These methods involve combinations of activation functions, sampling steps and different kinds of penalties. To investigate the effectiveness of sparsity by itself, we propose the k-sparse autoencoder, which is an autoencoder with linear activation function, where in hidden layers only the k highest activities are kept. When applied to the MNIST and NORB datasets, we find that this method achieves better classification results than denoising autoencoders, networks trained with dropout, and RBMs. k-sparse autoencoders are simple to train and the encoding stage is very fast, making them well-suited to large problem sizes, where conventional sparse coding algorithms cannot be applied.

References in corpus (2)

• Improving neural networks by preventing co-adaptation of feature detectors
• Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition

Cited by in corpus (43)

• Deep Learning in Alzheimer's disease: Diagnostic Classification and Prognostic Prediction using Neuroimaging Data
• MARTA GANs: Unsupervised Representation Learning for Remote Sensing Image Classification
• Stacked What-Where Auto-encoders
• Deep Learning of Part-based Representation of Data Using Sparse Autoencoders with Nonnegativity Constraints
• Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning
• Review: Deep Learning in Electron Microscopy
• RecGURU: Adversarial Learning of Generalized User Representations for Cross-Domain Recommendation
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• SoftAdapt: Techniques for Adaptive Loss Weighting of Neural Networks with Multi-Part Loss Functions
• How Can We Be So Dense? The Benefits of Using Highly Sparse Representations
• Sparse Unsupervised Capsules Generalize Better
• ACCNet: Actor-Coordinator-Critic Net for "Learning-to-Communicate" with Deep Multi-agent Reinforcement Learning
• Zero-bias autoencoders and the benefits of co-adapting features
• Toward Decoding the Relationship between Domain Structure and Functionality in Ferroelectrics via Hidden Latent Variables
• Greedy Deep Dictionary Learning
• A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks
• xSense: Learning Sense-Separated Sparse Representations and Textual Definitions for Explainable Word Sense Networks
• Deep learning models for predictive maintenance: a survey, comparison, challenges and prospect
• A PCA-like Autoencoder
• Deep Reasoning Networks: Thinking Fast and Slow
• Sparsely Activated Networks
• Author2Vec: A Framework for Generating User Embedding
• The Utility of Sparse Representations for Control in Reinforcement Learning
• Towards a Near Universal Time Series Data Mining Tool: Introducing the Matrix Profile
• Sparse Autoencoder for Unsupervised Nucleus Detection and Representation in Histopathology Images
• Sparse Factorization Layers for Neural Networks with Limited Supervision
• Quadratic Autoencoder (Q-AE) for Low-dose CT Denoising
• Learning Simple Thresholded Features with Sparse Support Recovery
• SCAT: Second Chance Autoencoder for Textual Data
• Fault-Diagnosing SLAM for Varying Scale Change Detection
• Soft Autoencoder and Its Wavelet Adaptation Interpretation
• Learning and Evaluating Sparse Interpretable Sentence Embeddings
• Compressive Features in Offline Reinforcement Learning for Recommender Systems
• Transform Invariant Auto-encoder
• Representation Learning by Reconstructing Neighborhoods
• End-To-End Dilated Variational Autoencoder with Bottleneck Discriminative Loss for Sound Morphing -- A Preliminary Study
• A Neuro-Inspired Autoencoding Defense Against Adversarial Perturbations
• Metalearning: Sparse Variable-Structure Automata
• Efficient Sparse Artificial Neural Networks
• Evaluating Sparse Interpretable Word Embeddings for Biomedical Domain
• Unsupervised Representation Learning with Laplacian Pyramid Auto-encoders
• Inducing Hierarchical Compositional Model by Sparsifying Generator Network
• Learning sparse transformations through backpropagation