Variations on the Chebyshev-Lagrange Activation Function

Abstract

We seek to improve the data efficiency of neural networks and present novel implementations of parameterized piece-wise polynomial activation functions. The parameters are the y-coordinates of n+1 Chebyshev nodes per hidden unit and Lagrangian interpolation between the nodes produces the polynomial on [-1, 1]. We show results for different methods of handling inputs outside [-1, 1] on synthetic datasets, finding significant improvements in capacity of expression and accuracy of interpolation in models that compute some form of linear extrapolation from either ends. We demonstrate competitive or state-of-the-art performance on the classification of images (MNIST and CIFAR-10) and minimally-correlated vectors (DementiaBank) when we replace ReLU or tanh with linearly extrapolated Chebyshev-Lagrange activations in deep residual architectures.

References in corpus (8)

• Shake-Shake regularization
• Polynomial Regression As an Alternative to Neural Nets
• Spectral Multigraph Networks for Discovering and Fusing Relationships in Molecules
• On the importance of normative data in speech-based assessment
• The Effect of Heterogeneous Data for Alzheimer's Disease Detection from Speech
• Semi-supervised classification by reaching consensus among modalities
• DeepConsensus: using the consensus of features from multiple layers to attain robust image classification
• Discontinuous Piecewise Polynomial Neural Networks