Self-Normalizing Neural Networks

Abstract

Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are "scaled exponential linear units" (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance -- even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization, and (3) to make learning highly robust. Furthermore, for activations not close to unit variance, we prove an upper and lower bound on the variance, thus, vanishing and exploding gradients are impossible. We compared SNNs on (a) 121 tasks from the UCI machine learning repository, on (b) drug discovery benchmarks, and on (c) astronomy tasks with standard FNNs and other machine learning methods such as random forests and support vector machines. SNNs significantly outperformed all competing FNN methods at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and set a new record at an astronomy data set. The winning SNN architectures are often very deep. Implementations are available at: github.com/bioinf-jku/SNNs.

9 pages (+ 93 pages appendix)

Cited by in corpus (20)

• Methods for Interpreting and Understanding Deep Neural Networks
• Training Deep AutoEncoders for Collaborative Filtering
• Training Neural Response Selection for Task-Oriented Dialogue Systems
• Multi-Generator Generative Adversarial Nets
• Improving Deep Learning by Inverse Square Root Linear Units (ISRLUs)
• Automated Cloud Provisioning on AWS using Deep Reinforcement Learning
• Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields
• Regularizing Activation Distribution for Training Binarized Deep Networks
• Predicting Adolescent Suicide Attempts with Neural Networks
• Learning to Run with Actor-Critic Ensemble
• Empirical analysis of non-linear activation functions for Deep Neural Networks in classification tasks
• Convolutional neural networks for structured omics: OmicsCNN and the OmicsConv layer
• MinCall - MinION end2end convolutional deep learning basecaller
• NIPS - Not Even Wrong? A Systematic Review of Empirically Complete Demonstrations of Algorithmic Effectiveness in the Machine Learning and Artificial Intelligence Literature
• Learning Compact Neural Networks Using Ordinary Differential Equations as Activation Functions
• Gaussian Process Neurons Learn Stochastic Activation Functions
• Solving internal covariate shift in deep learning with linked neurons
• Disentangling Video with Independent Prediction
• Hide-and-Tell: Learning to Bridge Photo Streams for Visual Storytelling
• Supervised Domain Enablement Attention for Personalized Domain Classification