Activation Functions: Comparison of trends in Practice and Research for Deep Learning

Abstract

Deep neural networks have been successfully used in diverse emerging domains to solve real world complex problems with may more deep learning(DL) architectures, being developed to date. To achieve these state-of-the-art performances, the DL architectures use activation functions (AFs), to perform diverse computations between the hidden layers and the output layers of any given DL architecture. This paper presents a survey on the existing AFs used in deep learning applications and highlights the recent trends in the use of the activation functions for deep learning applications. The novelty of this paper is that it compiles majority of the AFs used in DL and outlines the current trends in the applications and usage of these functions in practical deep learning deployments against the state-of-the-art research results. This compilation will aid in making effective decisions in the choice of the most suitable and appropriate activation function for any given application, ready for deployment. This paper is timely because most research papers on AF highlights similar works and results while this paper will be the first, to compile the trends in AF applications in practice against the research results from literature, found in deep learning research to date.

arXiv comment20 pages, 6 figures, 2 tables

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Deep Networks with Stochastic Depth
• Comparison of non-linear activation functions for deep neural networks on MNIST classification task

Cited by in corpus (89)

• Recent Advances and Applications of Deep Learning Methods in Materials Science
• A survey on modern trainable activation functions
• A Review on Deep Learning in UAV Remote Sensing
• GSA-DenseNet121-COVID-19: a Hybrid Deep Learning Architecture for the Diagnosis of COVID-19 Disease based on Gravitational Search Optimization Algorithm
• Quantum agents in the Gym: a variational quantum algorithm for deep Q-learning
• A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU
• Review and Comparison of Commonly Used Activation Functions for Deep Neural Networks
• Review: Deep Learning in Electron Microscopy
• Discovering Parametric Activation Functions
• SPECULATOR: Emulating stellar population synthesis for fast and accurate galaxy spectra and photometry
• Handwritten Digit Recognition using Machine and Deep Learning Algorithms
• DPFM: Deep Partial Functional Maps
• ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
• Generative Modeling of Turbulence
• Rapid Identification of Strongly Lensed Gravitational-Wave Events with Machine Learning
• $\text{FastSVD-ML-ROM}$: A Reduced-Order Modeling Framework based on Machine Learning for Real-Time Applications
• Anomalous Jet Identification via Sequence Modeling
• Improved deep learning techniques in gravitational-wave data analysis
• Adaptive estimation of quantum observables
• Solving Allen-Cahn and Cahn-Hilliard Equations using the Adaptive Physics Informed Neural Networks
• Galaxy classification: a deep learning approach for classifying Sloan Digital Sky Survey images
• Mass Estimation of Galaxy Clusters with Deep Learning II: CMB Cluster Lensing
• DeepSentiPers: Novel Deep Learning Models Trained Over Proposed Augmented Persian Sentiment Corpus
• Software Vulnerability Detection via Deep Learning over Disaggregated Code Graph Representation
• Deep-CEE I: Fishing for Galaxy Clusters with Deep Neural Nets
• Multi-limb Split Learning for Tumor Classification on Vertically Distributed Data
• SCSS-Net: Solar Corona Structures Segmentation by Deep Learning
• The Principle of Minimum Pressure Gradient: An Alternative Basis for Physics-Informed Learning of Incompressible Fluid Mechanics
• Effects of the Nonlinearity in Activation Functions on the Performance of Deep Learning Models
• Modelling parametric uncertainty in PDEs models via Physics-Informed Neural Networks
• A Survey on Activation Functions and their relation with Xavier and He Normal Initialization
• Effect of Activation Functions on the Training of Overparametrized Neural Nets
• Deep learning insights into cosmological structure formation
• Prediction Model for Mortality Analysis of Pregnant Women Affected With COVID-19
• Machine learning synthetic spectra for probabilistic redshift estimation: SYTH-Z
• Separation of track- and shower-like energy deposits in ProtoDUNE-SP using a convolutional neural network
• Improved Cardinality Estimation by Learning Queries Containment Rates
• An astronomical image content-based recommendation system using combined deep learning models in a fully unsupervised mode
• Non-intrusive Surrogate Modeling for Parametrized Time-dependent PDEs using Convolutional Autoencoders
• Bridging the Gap: Machine Learning to Resolve Improperly Modeled Dynamics
• Neural Networks with A La Carte Selection of Activation Functions
• On the Implementation of Fixed-point Exponential Function for Machine Learning and Signal Processing Accelerators
• Knowledge-based Radiation Treatment Planning: A Data-driven Method Survey
• Smooth activations and reproducibility in deep networks
• Residual-Concatenate Neural Network with Deep Regularization Layers for Binary Classification
• Activation function impact on Sparse Neural Networks
• Beyond Accuracy: An Empirical Study on Unit Testing in Open-source Deep Learning Projects
• Growing Cosine Unit: A Novel Oscillatory Activation Function That Can Speedup Training and Reduce Parameters in Convolutional Neural Networks
• E(n) Equivariant Normalizing Flows
• Model Complexity of Deep Learning: A Survey
• Effect of the output activation function on the probabilities and errors in medical image segmentation
• Towards Label-Free 3D Segmentation of Optical Coherence Tomography Images of the Optic Nerve Head Using Deep Learning
• TanhSoft -- a family of activation functions combining Tanh and Softplus
• Learning from Videos with Deep Convolutional LSTM Networks
• Deep Residual Neural Networks for Image in Speech Steganography
• Measuring Model Complexity of Neural Networks with Curve Activation Functions
• Automated Diabetic Retinopathy Grading using Deep Convolutional Neural Network
• Modelling of daily reference evapotranspiration using deep neural network in different climates
• Evading Malware Classifiers via Monte Carlo Mutant Feature Discovery
• Model order reduction with neural networks: Application to laminar and turbulent flows
• Real-Time Forecasting of Dockless Scooter-Sharing Demand: A Spatio-Temporal Multi-Graph Transformer Approach
• Detection of False-Reading Attacks in the AMI Net-Metering System
• SMYRF: Efficient Attention using Asymmetric Clustering
• Learn-able parameter guided Activation Functions
• Measuring Performance of Generative Adversarial Networks on Devanagari Script
• The Seven-League Scheme: Deep learning for large time step Monte Carlo simulations of stochastic differential equations
• MG-GCN: Fast and Effective Learning with Mix-grained Aggregators for Training Large Graph Convolutional Networks
• RealSmileNet: A Deep End-To-End Network for Spontaneous and Posed Smile Recognition
• Optimizations of Autoencoders for Analysis and Classification of Microscopic In Situ Hybridization Images
• Performance of artificial neural networks in an inverse problem of laser beam diagnostics
• Hyperspectral imaging with Raman scattered photons: A new paradigm in Raman analysis
• Distribution Mismatch Correction for Improved Robustness in Deep Neural Networks
• A Generative Model based Adversarial Security of Deep Learning and Linear Classifier Models
• Competition analysis on the over-the-counter credit default swap market
• Construction of a far ultraviolet all sky map from an incomplete survey: Application of a deep learning algorithm
• NN-based Transformation of Any SQL Cardinality Estimator for Handling DISTINCT, AND, OR and NOT
• Overcoming Overfitting and Large Weight Update Problem in Linear Rectifiers: Thresholded Exponential Rectified Linear Units
• Using neural networks to predict icephobic performance
• Approximate Wireless Communication for Lossy Gradient Updates in IoT Federated Learning
• Neural network emulator to constrain the high-$z$ IGM thermal state from Lyman-$\alpha$ forest flux auto-correlation function
• WisdomNet: Prognosis of COVID-19 with Slender Prospect of False Negative Cases and Vaticinating the Probability of Maturation to ARDS using Posteroanterior Chest X-Rays
• OpTorch: Optimized deep learning architectures for resource limited environments
• A new smart-cropping pipeline for prostate segmentation using deep learning networks
• GIM: Gaussian Isolation Machines
• IC Neuron: An Efficient Unit to Construct Neural Networks
• A survey on deep learning approaches for breast cancer diagnosis
• LaNet: Real-time Lane Identification by Learning Road SurfaceCharacteristics from Accelerometer Data
• Privacy-Preserving and Efficient Data Collection Scheme for AMI Networks Using Deep Learning
• Efficient Hardware Realizations of Feedforward Artificial Neural Networks