Comparison of non-linear activation functions for deep neural networks on MNIST classification task

Abstract

Activation functions play a key role in neural networks so it becomes fundamental to understand their advantages and disadvantages in order to achieve better performances. This paper will first introduce common types of non linear activation functions that are alternative to the well known sigmoid function and then evaluate their characteristics. Moreover deeper neural networks will be analysed because they positively influence the final performances compared to shallower networks. They also strictly depend on the weight initialisation hence the effect of drawing weights from Gaussian and uniform distribution will be analysed making particular attention on how the number of incoming and outgoing connection to a node influence the whole network.

arXiv comment5 pages, 6 figures

References in corpus (2)

• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Self-Normalizing Neural Networks

Cited by in corpus (17)

• Activation Functions: Comparison of trends in Practice and Research for Deep Learning
• A survey on modern trainable activation functions
• Nonparametric regression using deep neural networks with ReLU activation function
• Review: Deep Learning in Electron Microscopy
• A deep learning approach to cosmological dark energy models
• QReLU and m-QReLU: Two novel quantum activation functions to aid medical diagnostics
• A simple and efficient architecture for trainable activation functions
• A Survey on Machine Learning Techniques for Auto Labeling of Video, Audio, and Text Data
• Matrix Element Regression with Deep Neural Networks -- breaking the CPU barrier
• Effects of the Nonlinearity in Activation Functions on the Performance of Deep Learning Models
• A Methodology for Automatic Selection of Activation Functions to Design Hybrid Deep Neural Networks
• Multivariate Forecasting of Crude Oil Spot Prices using Neural Networks
• Inversion of Integral Models: a Neural Network Approach
• MILR: Mathematically Induced Layer Recovery for Plaintext Space Error Correction of CNNs
• Central and Non-central Limit Theorems arising from the Scattering Transform and its Neural Activation Generalization
• A new neural-network-based model for measuring the strength of a pseudorandom binary sequence
• Joint Model and Data Driven Receiver Design for Data-Dependent Superimposed Training Scheme with Imperfect Hardware