EIS -- a family of activation functions combining Exponential, ISRU, and Softplus

Abstract

Activation functions play a pivotal role in the function learning using neural networks. The non-linearity in the learned function is achieved by repeated use of the activation function. Over the years, numerous activation functions have been proposed to improve accuracy in several tasks. Basic functions like ReLU, Sigmoid, Tanh, or Softplus have been favorite among the deep learning community because of their simplicity. In recent years, several novel activation functions arising from these basic functions have been proposed, which have improved accuracy in some challenging datasets. We propose a five hyper-parameters family of activation functions, namely EIS, defined as, \[ \frac{x(\ln(1+e^x))^α}{\sqrt{β+γx^2}+δe^{-θx}}. \] We show examples of activation functions from the EIS family which outperform widely used activation functions on some well known datasets and models. For example, $\frac{x\ln (1+e^x)}{x+1.16e^{-x}}$ beats ReLU by 0.89\% in DenseNet-169, 0.24\% in Inception V3 in CIFAR100 dataset while 1.13\% in Inception V3, 0.13\% in DenseNet-169, 0.94\% in SimpleNet model in CIFAR10 dataset. Also, $\frac{x\ln (1+e^x)}{\sqrt{1+x^2}}$ beats ReLU by 1.68\% in DenseNet-169, 0.30\% in Inception V3 in CIFAR100 dataset while 1.0\% in Inception V3, 0.15\% in DenseNet-169, 1.13\% in SimpleNet model in CIFAR10 dataset.

arXiv comment10 pages

References in corpus (10)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Gaussian Error Linear Units (GELUs)
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Searching for Activation Functions
• DeepArchitect: Automatically Designing and Training Deep Architectures
• Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures
• Improving Deep Learning by Inverse Square Root Linear Units (ISRLUs)
• TanhSoft -- a family of activation functions combining Tanh and Softplus