q-Neurons: Neuron Activations based on Stochastic Jackson's Derivative Operators

Abstract

We propose a new generic type of stochastic neurons, called $q$-neurons, that considers activation functions based on Jackson's $q$-derivatives with stochastic parameters $q$. Our generalization of neural network architectures with $q$-neurons is shown to be both scalable and very easy to implement. We demonstrate experimentally consistently improved performances over state-of-the-art standard activation functions, both on training and testing loss functions.

12 pages, 5 figures, 1 table

References in corpus (5)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Gaussian Error Linear Units (GELUs)
• Learning Activation Functions to Improve Deep Neural Networks
• Better Approximations of High Dimensional Smooth Functions by Deep Neural Networks with Rectified Power Units
• Enhanced $q$-Least Mean Square