A simple and efficient architecture for trainable activation functions
arXiv:1902.03306 · doi:10.1016/j.neucom.2019.08.065
Abstract
Learning automatically the best activation function for the task is an active topic in neural network research. At the moment, despite promising results, it is still difficult to determine a method for learning an activation function that is at the same time theoretically simple and easy to implement. Moreover, most of the methods proposed so far introduce new parameters or adopt different learning techniques. In this work we propose a simple method to obtain trained activation function which adds to the neural network local subnetworks with a small amount of neurons. Experiments show that this approach could lead to better result with respect to using a pre-defined activation function, without introducing a large amount of extra parameters that need to be learned.
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- Self-Normalizing Neural Networks
- Learning Activation Functions to Improve Deep Neural Networks
- An ensemble-based system for automatic screening of diabetic retinopathy
- Neural Network with Unbounded Activation Functions is Universal Approximator
- The Power of Depth for Feedforward Neural Networks
- Comparison of non-linear activation functions for deep neural networks on MNIST classification task
- Deep Learning with S-shaped Rectified Linear Activation Units
- A single hidden layer feedforward network with only one neuron in the hidden layer can approximate any univariate function
- Activation Ensembles for Deep Neural Networks
Cited by in corpus (5)
- A survey on modern trainable activation functions
- Curvature-based Feature Selection with Application in Classifying Electronic Health Records
- Learning the Proximity Operator in Unfolded ADMM for Phase Retrieval
- Learning Neural Activations
- Robust Deep Network Learning of Nonlinear Regression Tasks by Parametric Leaky Exponential Linear Units (LELUs) and a Diffusion Metric