Self-Normalizing Neural Networks

Abstract

Deep Learning has revolutionized vision via convolutional neural networks (CNNs) and natural language processing via recurrent neural networks (RNNs). However, success stories of Deep Learning with standard feed-forward neural networks (FNNs) are rare. FNNs that perform well are typically shallow and, therefore cannot exploit many levels of abstract representations. We introduce self-normalizing neural networks (SNNs) to enable high-level abstract representations. While batch normalization requires explicit normalization, neuron activations of SNNs automatically converge towards zero mean and unit variance. The activation function of SNNs are "scaled exponential linear units" (SELUs), which induce self-normalizing properties. Using the Banach fixed-point theorem, we prove that activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance -- even under the presence of noise and perturbations. This convergence property of SNNs allows to (1) train deep networks with many layers, (2) employ strong regularization, and (3) to make learning highly robust. Furthermore, for activations not close to unit variance, we prove an upper and lower bound on the variance, thus, vanishing and exploding gradients are impossible. We compared SNNs on (a) 121 tasks from the UCI machine learning repository, on (b) drug discovery benchmarks, and on (c) astronomy tasks with standard FNNs and other machine learning methods such as random forests and support vector machines. SNNs significantly outperformed all competing FNN methods at 121 UCI tasks, outperformed all competing methods at the Tox21 dataset, and set a new record at an astronomy data set. The winning SNN architectures are often very deep. Implementations are available at: github.com/bioinf-jku/SNNs.

9 pages (+ 93 pages appendix)

Cited by in corpus (114)

• Methods for Interpreting and Understanding Deep Neural Networks
• DeepSurv: Personalized Treatment Recommender System Using A Cox Proportional Hazards Deep Neural Network
• Over the Air Deep Learning Based Radio Signal Classification
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Attention in Natural Language Processing
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Deep 1D-Convnet for accurate Parkinson disease detection and severity prediction from gait
• Single-Image SVBRDF Capture with a Rendering-Aware Deep Network
• Unveiling the potential of Graph Neural Networks for network modeling and optimization in SDN
• Deep learning extends de novo protein modelling coverage of genomes using iteratively predicted structural constraints
• EddyNet: A Deep Neural Network For Pixel-Wise Classification of Oceanic Eddies
• Learning Constitutive Relations from Indirect Observations Using Deep Neural Networks
• Machine Learning in QM/MM Molecular Dynamics Simulations of Condensed-Phase Systems
• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data
• Ensemble Kalman Inversion: A Derivative-Free Technique For Machine Learning Tasks
• Deep Learning in Mobile and Wireless Networking: A Survey
• Deep Chain HDRI: Reconstructing a High Dynamic Range Image from a Single Low Dynamic Range Image
• Comparison of non-linear activation functions for deep neural networks on MNIST classification task
• A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data
• Biology and medicine in the landscape of quantum advantages
• VIBNN: Hardware Acceleration of Bayesian Neural Networks
• Efficient Hyperparameter Optimization in Deep Learning Using a Variable Length Genetic Algorithm
• Deep learning prediction of patient response time course from early data via neural-pharmacokinetic/pharmacodynamic modeling
• QReLU and m-QReLU: Two novel quantum activation functions to aid medical diagnostics
• SMILES2Vec: An Interpretable General-Purpose Deep Neural Network for Predicting Chemical Properties
• Training Deep AutoEncoders for Collaborative Filtering
• Training Neural Response Selection for Task-Oriented Dialogue Systems
• Multi-Generator Generative Adversarial Nets
• Automatic detection and decoding of honey bee waggle dances
• Convolutional Neural Networks for Continuous QoE Prediction in Video Streaming Services
• Activation Functions in Artificial Neural Networks: A Systematic Overview
• On the Selection of Initialization and Activation Function for Deep Neural Networks
• Improving Deep Learning by Inverse Square Root Linear Units (ISRLUs)
• A simple and efficient architecture for trainable activation functions
• Understanding Batch Normalization
• PHD-GIFs: Personalized Highlight Detection for Automatic GIF Creation
• Automated Cloud Provisioning on AWS using Deep Reinforcement Learning
• Coulomb GANs: Provably Optimal Nash Equilibria via Potential Fields
• The DNNLikelihood: enhancing likelihood distribution with Deep Learning
• A comparison of optimisation algorithms for high-dimensional particle and astrophysics applications
• Unsupervised Multimodal Representation Learning across Medical Images and Reports
• Group Convolutional Neural Networks Improve Quantum State Accuracy
• EENMF: An End-to-End Neural Matching Framework for E-Commerce Sponsored Search
• Multi-Modal Adversarial Autoencoders for Recommendations of Citations and Subject Labels
• Regularizing Activation Distribution for Training Binarized Deep Networks
• Reconstructing Patchy Reionization with Deep Learning
• Unsupervised Clustering of Quantitative Imaging Phenotypes using Autoencoder and Gaussian Mixture Model
• Predicting Adolescent Suicide Attempts with Neural Networks
• Learning to Run with Actor-Critic Ensemble
• GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning
• Empirical analysis of non-linear activation functions for Deep Neural Networks in classification tasks
• Bayesian Inference for Large Scale Image Classification
• A Methodology for Automatic Selection of Activation Functions to Design Hybrid Deep Neural Networks
• Reconstructing Cosmic Polarization Rotation with ResUNet-CMB
• Normalization of Neural Networks using Analytic Variance Propagation
• Breaking the Activation Function Bottleneck through Adaptive Parameterization
• Feedback-Based Tree Search for Reinforcement Learning
• Convolutional neural networks for structured omics: OmicsCNN and the OmicsConv layer
• The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks
• Sensitivity - Local Index to Control Chaoticity or Gradient Globally -
• Equilibrium Learning in Combinatorial Auctions: Computing Approximate Bayesian Nash Equilibria via Pseudogradient Dynamics
• MinCall - MinION end2end convolutional deep learning basecaller
• An Attention-Gated Convolutional Neural Network for Sentence Classification
• Learning Compact Neural Networks Using Ordinary Differential Equations as Activation Functions
• Automated Copper Alloy Grain Size Evaluation Using a Deep-learning CNN
• A Sober Look at Neural Network Initializations
• NIPS - Not Even Wrong? A Systematic Review of Empirically Complete Demonstrations of Algorithmic Effectiveness in the Machine Learning and Artificial Intelligence Literature
• A Streamlined Encoder/Decoder Architecture for Melody Extraction
• Power Consumption Variation over Activation Functions
• Learning Context-Dependent Choice Functions
• Machine Learning for Mechanical Ventilation Control
• Genetic Neural Architecture Search for automatic assessment of human sperm images
• Gaussian Process Neurons Learn Stochastic Activation Functions
• SGR: Self-Supervised Spectral Graph Representation Learning
• Simulating Continuum Mechanics with Multi-Scale Graph Neural Networks
• Solving internal covariate shift in deep learning with linked neurons
• Robust and Scalable Differentiable Neural Computer for Question Answering
• Interpretable Visualizations with Differentiating Embedding Networks
• Modelling of daily reference evapotranspiration using deep neural network in different climates
• A Scalable Neural Shortlisting-Reranking Approach for Large-Scale Domain Classification in Natural Language Understanding
• Shapley Interpretation and Activation in Neural Networks
• Generative Adversarial User Privacy in Lossy Single-Server Information Retrieval
• Semi-Supervised Variational Autoencoder for Survival Prediction
• Duodepth: Static Gesture Recognition Via Dual Depth Sensors
• Generative Visual Rationales
• Efficient Large-Scale Domain Classification with Personalized Attention
• Unsupervised Emergence of Spatial Structure from Sensorimotor Prediction
• Anomaly Detection by Recombining Gated Unsupervised Experts
• Fréchet ChemNet Distance: A metric for generative models for molecules in drug discovery
• Convolutional Neural Networks to Enhance Coded Speech
• Argumentative Relation Classification as Plausibility Ranking
• Modality Attention for End-to-End Audio-visual Speech Recognition
• Inverse design of multilayer nanoparticles using artificial neural networks and genetic algorithm
• Generalized Batch Normalization: Towards Accelerating Deep Neural Networks
• A New Multifractal-based Deep Learning Model for Text Mining
• DELMU: A Deep Learning Approach to Maximising the Utility of Virtualised Millimetre-Wave Backhauls
• Dropout with Tabu Strategy for Regularizing Deep Neural Networks
• Deep learning improved by biological activation functions
• Detection of Glottal Closure Instants from Raw Speech using Convolutional Neural Networks
• Volatility model calibration with neural networks a comparison between direct and indirect methods
• Hide-and-Tell: Learning to Bridge Photo Streams for Visual Storytelling
• Disentangling Video with Independent Prediction
• D-PCN: Parallel Convolutional Networks for Image Recognition via a Discriminator
• An Iterative Closest Points Approach to Neural Generative Models
• Sorted Pooling in Convolutional Networks for One-shot Learning
• Aggregation of binary feature descriptors for compact scene model representation in large scale structure-from-motion applications
• Supervised Domain Enablement Attention for Personalized Domain Classification
• Robust Implicit Backpropagation
• Optimizing generalization on the train set: a novel gradient-based framework to train parameters and hyperparameters simultaneously
• MetaPose: Fast 3D Pose from Multiple Views without 3D Supervision
• Frame-level SpecAugment for Deep Convolutional Neural Networks in Hybrid ASR Systems
• ConFuse: Convolutional Transform Learning Fusion Framework For Multi-Channel Data Analysis
• SConE: Siamese Constellation Embedding Descriptor for Image Matching
• Target Model Agnostic Adversarial Attacks with Query Budgets on Language Understanding Models