The Role of Neural Network Activation Functions

Abstract

A wide variety of activation functions have been proposed for neural networks. The Rectified Linear Unit (ReLU) is especially popular today. There are many practical reasons that motivate the use of the ReLU. This paper provides new theoretical characterizations that support the use of the ReLU, its variants such as the leaky ReLU, as well as other activation functions in the case of univariate, single-hidden layer feedforward neural networks. Our results also explain the importance of commonly used strategies in the design and training of neural networks such as "weight decay" and "path-norm" regularization, and provide a new justification for the use of "skip connections" in network architectures. These new insights are obtained through the lens of spline theory. In particular, we show how neural network training problems are related to infinite-dimensional optimizations posed over Banach spaces of functions whose solutions are well-known to be fractional and polynomial splines, where the particular Banach space (which controls the order of the spline) depends on the choice of activation function.

arXiv commentupdate to published version

References in corpus (10)

• Exploring Generalization in Deep Learning
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
• To understand deep learning we need to understand kernel learning
• Splines are Universal Solutions of Linear Inverse Problems with Generalized-TV regularization
• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
• Does data interpolation contradict statistical optimality?
• Continuous-Domain Solutions of Linear Inverse Problems with Tikhonov vs. Generalized TV Regularization
• Banach Space Representer Theorems for Neural Networks and Ridge Splines
• A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case

Cited by in corpus (11)

• What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory
• Banach Space Representer Theorems for Neural Networks and Ridge Splines
• Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks
• Deep Learning Meets Sparse Regularization: A Signal Processing Perspective
• Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularization
• Revealing the Structure of Deep Neural Networks via Convex Duality
• Convex Geometry and Duality of Over-parameterized Neural Networks
• Analysis on reservoir activation with the nonlinearity harnessed from solution-processed molybdenum disulfide
• Controlled Learning of Pointwise Nonlinearities in Neural-Network-Like Architectures
• Function-Space Optimality of Neural Architectures with Multivariate Nonlinearities
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks