Learning Activation Functions to Improve Deep Neural Networks

Abstract

Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural network architectures composed of static rectified linear units, achieving state-of-the-art performance on CIFAR-10 (7.51%), CIFAR-100 (30.83%), and a benchmark from high-energy physics involving Higgs boson decay modes.

Accepted as a workshop paper contribution at the International Conference on Learning Representations (ICLR) 2015

References in corpus (7)

• Improving neural networks by preventing co-adaptation of feature detectors
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Searching for Exotic Particles in High-Energy Physics with Deep Learning
• Deeply-Supervised Nets
• Deep Networks with Internal Selective Attention through Feedback Connections
• Enhanced Higgs to ${\tau }^{+}{\tau }^{-}$ Searches with Deep Learning
• From Maxout to Channel-Out: Encoding Information on Sparse Pathways

Cited by in corpus (72)

• Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Searching for Activation Functions
• Optimization Problems for Machine Learning: A Survey
• Convolutional neural networks with low-rank regularization
• Systematic evaluation of CNN advances on the ImageNet
• Deep Networks with Stochastic Depth
• Shallow Neural Networks for Fluid Flow Reconstruction with Limited Sensors
• Federated Learning in Mobile Edge Networks: A Comprehensive Survey
• Deep Residual Networks with Exponential Linear Unit
• Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree
• Review: Deep Learning in Electron Microscopy
• Deep Learning with S-shaped Rectified Linear Activation Units
• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
• Discovering Parametric Activation Functions
• Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
• Learning Combinations of Activation Functions
• Efficient Continual Learning in Neural Networks with Embedding Regularization
• A simple and efficient architecture for trainable activation functions
• Learning Specialized Activation Functions for Physics-informed Neural Networks
• Doubly Convolutional Neural Networks
• Deep Neural Networks with Trainable Activations and Controlled Lipschitz Constant
• All-optical nonlinear activation function based on stimulated Brillouin scattering
• Soft-Root-Sign Activation Function
• Funnel Activation for Visual Recognition
• AReLU: Attention-based Rectified Linear Unit
• Neural Networks with Smooth Adaptive Activation Functions for Regression
• Activation Ensembles for Deep Neural Networks
• Frank-Wolfe Network: An Interpretable Deep Structure for Non-Sparse Coding
• A Survey on Activation Functions and their relation with Xavier and He Normal Initialization
• A Probabilistic Framework for Nonlinearities in Stochastic Neural Networks
• Activate or Not: Learning Customized Activation
• Deep Learning applied to Road Traffic Speed forecasting
• Neural Networks with A La Carte Selection of Activation Functions
• CIFAR-10: KNN-based Ensemble of Classifiers
• Multi-bin Trainable Linear Unit for Fast Image Restoration Networks
• Faster Convergence in Deep-Predictive-Coding Networks to Learn Deeper Representations
• Generate High Resolution Images With Generative Variational Autoencoder
• Learning Compact Neural Networks Using Ordinary Differential Equations as Activation Functions
• Cascaded Subpatch Networks for Effective CNNs
• Collaborative Layer-wise Discriminative Learning in Deep Neural Networks
• Gaussian Process Neurons Learn Stochastic Activation Functions
• Generating Magnetic Resonance Spectroscopy Imaging Data of Brain Tumours from Linear, Non-Linear and Deep Learning Models
• A representer theorem for deep neural networks
• Combinatorially Generated Piecewise Activation Functions
• Piecewise Linear Units Improve Deep Neural Networks
• Learning Neuron Non-Linearities with Kernel-Based Deep Neural Networks
• DeepLABNet: End-to-end Learning of Deep Radial Basis Networks with Fully Learnable Basis Functions
• q-Neurons: Neuron Activations based on Stochastic Jackson's Derivative Operators
• Hardware-Driven Nonlinear Activation for Stochastic Computing Based Deep Convolutional Neural Networks
• Evolving Parsimonious Networks by Mixing Activation Functions
• Comparisons among different stochastic selection of activation layers for convolutional neural networks for healthcare
• Hyperparameter Transfer Across Developer Adjustments
• Connections Between Pairs of Filters Improve the Accuracy of Convolutional Neural Networks
• Deep Asymmetric Networks with a Set of Node-wise Variant Activation Functions
• Improving neural networks with bunches of neurons modeled by Kumaraswamy units: Preliminary study
• Overcoming Overfitting and Large Weight Update Problem in Linear Rectifiers: Thresholded Exponential Rectified Linear Units
• Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions
• Machine Learning Approach for Transforming Scattering Parameters to Complex Permittivity
• D-PCN: Parallel Convolutional Networks for Image Recognition via a Discriminator
• Regularized Flexible Activation Function Combinations for Deep Neural Networks
• Convolution in Convolution for Network in Network
• Progressive VAE Training on Highly Sparse and Imbalanced Data
• Tensor-based framework for training flexible neural networks
• Neurons Activation Visualization and Information Theoretic Analysis
• Light Multi-segment Activation for Model Compression
• On the Stability and Generalization of Learning with Kernel Activation Functions
• Scalable Partial Explainability in Neural Networks via Flexible Activation Functions
• Graph Interpolating Activation Improves Both Natural and Robust Accuracies in Data-Efficient Deep Learning
• An Effective Training Method For Deep Convolutional Neural Network
• Goldilocks Neural Networks
• PolyNeuron: Automatic Neuron Discovery via Learned Polyharmonic Spline Activations