Goldilocks Neural Networks
arXiv:2002.05059
Abstract
We introduce the new "Goldilocks" class of activation functions, which non-linearly deform the input signal only locally when the input signal is in the appropriate range. The small local deformation of the signal enables better understanding of how and why the signal is transformed through the layers. Numerical results on CIFAR-10 and CIFAR-100 data sets show that Goldilocks networks perform better than, or comparably to SELU and RELU, while introducing tractability of data deformation through the layers.
References in corpus (7)
- Searching for Activation Functions
- Learning Activation Functions to Improve Deep Neural Networks
- Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning
- Random Walk Initialization for Training Very Deep Feedforward Networks
- Improving Deep Learning by Inverse Square Root Linear Units (ISRLUs)
- Shifting Mean Activation Towards Zero with Bipolar Activation Functions
- Training Deep Fourier Neural Networks To Fit Time-Series Data