Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks

Abstract

The performance of deep network learning strongly depends on the choice of the non-linear activation function associated with each neuron. However, deciding on the best activation is non-trivial, and the choice depends on the architecture, hyper-parameters, and even on the dataset. Typically these activations are fixed by hand before training. Here, we demonstrate how to eliminate the reliance on first picking fixed activation functions by using flexible parametric rational functions instead. The resulting Padé Activation Units (PAUs) can both approximate common activation functions and also learn new ones while providing compact representations. Our empirical evidence shows that end-to-end learning deep networks with PAUs can increase the predictive performance. Moreover, PAUs pave the way to approximations with provable robustness. https://github.com/ml-research/pau

arXiv comment17 Pages, 8 Figures

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Continuously Differentiable Exponential Linear Units
• Deep Learning with S-shaped Rectified Linear Activation Units
• Provable Robustness of ReLU networks via Maximization of Linear Regions
• Learning Combinations of Activation Functions
• Learning Activation Functions: A new paradigm for understanding Neural Networks

Cited by in corpus (9)

• Attention-Based Transformers for Instance Segmentation of Cells in Microstructures
• Universal Approximation with Deep Narrow Networks
• Effect of the output activation function on the probabilities and errors in medical image segmentation
• Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learning
• Orthogonal-Padé Activation Functions: Trainable Activation functions for smooth and faster convergence in deep networks
• Rational activation functions in neural networks with uniform based loss functions and its application in classification
• Deep Collective Learning: Learning Optimal Inputs and Weights Jointly in Deep Neural Networks
• Generation and Simulation of Yeast Microscopy Imagery with Deep Learning
• An Informal Introduction to Multiplet Neural Networks