Saturating Auto-Encoders

Abstract

We introduce a simple new regularizer for auto-encoders whose hidden-unit activation functions contain at least one zero-gradient (saturated) region. This regularizer explicitly encourages activations in the saturated region(s) of the corresponding activation function. We call these Saturating Auto-Encoders (SATAE). We show that the saturation regularizer explicitly limits the SATAE's ability to reconstruct inputs which are not near the data manifold. Furthermore, we show that a wide variety of features can be learned when different activation functions are used. Finally, connections are established with the Contractive and Sparse Auto-Encoders.

References in corpus (1)

• What Regularized Auto-Encoders Learn from the Data Generating Distribution

Cited by in corpus (8)

• Biologically inspired protection of deep networks from adversarial attacks
• Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networks
• Adversarial Examples - A Complete Characterisation of the Phenomenon
• Zero-bias autoencoders and the benefits of co-adapting features
• Unsupervised Feature Learning from Temporal Data
• Unsupervised Learning of Spatiotemporally Coherent Metrics
• Piecewise Linear Units Improve Deep Neural Networks
• Accelerating Training of Deep Neural Networks with a Standardization Loss