A survey on modern trainable activation functions

Abstract

In neural networks literature, there is a strong interest in identifying and defining activation functions which can improve neural network performance. In recent years there has been a renovated interest of the scientific community in investigating activation functions which can be trained during the learning process, usually referred to as "trainable", "learnable" or "adaptable" activation functions. They appear to lead to better network performance. Diverse and heterogeneous models of trainable activation function have been proposed in the literature. In this paper, we present a survey of these models. Starting from a discussion on the use of the term "activation function" in literature, we propose a taxonomy of trainable activation functions, highlight common and distinctive proprieties of recent and past models, and discuss main advantages and limitations of this type of approach. We show that many of the proposed approaches are equivalent to adding neuron layers which use fixed (non-trainable) activation functions and some simple local rule that constraints the corresponding weight layers.

arXiv commentPublished in "Neural Networks" journal (Elsevier)

References in corpus (19)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Activation Functions: Comparison of trends in Practice and Research for Deep Learning
• Searching for Exotic Particles in High-Energy Physics with Deep Learning
• Neural Network with Unbounded Activation Functions is Universal Approximator
• Comparison of non-linear activation functions for deep neural networks on MNIST classification task
• Deep Residual Networks with Exponential Linear Unit
• Enhanced Higgs to ${\tau }^{+}{\tau }^{-}$ Searches with Deep Learning
• Deep Learning with S-shaped Rectified Linear Activation Units
• Improving Deep Neural Networks with Probabilistic Maxout Units
• Batch-normalized Maxout Network in Network
• Revise Saturated Activation Functions
• A simple and efficient architecture for trainable activation functions
• Zero-bias autoencoders and the benefits of co-adapting features
• Flatten-T Swish: a thresholded ReLU-Swish-like activation function for deep learning
• E-swish: Adjusting Activations to Different Network Depths
• Gaussian Process Neurons Learn Stochastic Activation Functions

Cited by in corpus (25)

• Recent Advances in Deep Learning Techniques for Face Recognition
• Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey
• Exploiting auto-encoders and segmentation methods for middle-level explanations of image classification systems
• I-FENN with Temporal Convolutional Networks: expediting the load-history analysis of non-local gradient damage propagation
• Adaptive Activation Functions for Predictive Modeling with Sparse Experimental Data
• Non-linear Neurons with Human-like Apical Dendrite Activations
• A quantum collisional classifier driven by information reservoir
• Can physical information aid the generalization ability of Neural Networks for hydraulic modeling?
• Activation Function Optimization Scheme for Image Classification
• Energy reconstruction for large liquid scintillator detectors with machine learning techniques: aggregated features approach
• Biologically Inspired Oscillating Activation Functions Can Bridge the Performance Gap between Biological and Artificial Neurons
• Neural Networks with A La Carte Selection of Activation Functions
• Hidden Classification Layers: Enhancing linear separability between classes in neural networks layers
• Neuronal diversity can improve machine learning for physics and beyond
• A Bio-Inspired Chaos Sensor Model Based on the Perceptron Neural Network: Machine Learning Concept and Application for Computational Neuro-Science
• Effect of the output activation function on the probabilities and errors in medical image segmentation
• Generalized pair-wise logit dynamic and its connection to a mean field game: theoretical and computational investigations focusing on resource management
• Designing DNNs for a trade-off between robustness and processing performance in embedded devices
• Controlled Learning of Pointwise Nonlinearities in Neural-Network-Like Architectures
• Don't Fear Peculiar Activation Functions: EUAF and Beyond
• Multi-Activation Hidden Units for Neural Networks with Random Weights
• Robust Deep Network Learning of Nonlinear Regression Tasks by Parametric Leaky Exponential Linear Units (LELUs) and a Diffusion Metric
• Orthogonal Activation with Implicit Group-Aware Bias Learning for Class Imbalance
• Four Cubes
• Tensor-based framework for training flexible neural networks