Complexity Measures for Neural Networks with General Activation Functions Using Path-based Norms

Abstract

A simple approach is proposed to obtain complexity controls for neural networks with general activation functions. The approach is motivated by approximating the general activation functions with one-dimensional ReLU networks, which reduces the problem to the complexity controls of ReLU networks. Specifically, we consider two-layer networks and deep residual networks, for which path-based norms are derived to control complexities. We also provide preliminary analyses of the function spaces induced by these norms and a priori estimates of the corresponding regularized estimators.

47 pages

References in corpus (8)

• Gaussian Error Linear Units (GELUs)
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Searching for Activation Functions
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• A Priori Estimates of the Population Risk for Residual Networks
• Approximation and Estimation for High-Dimensional Deep Learning Networks
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• Global Capacity Measures for Deep ReLU Networks via Path Sampling

Cited by in corpus (5)

• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• Some observations on high-dimensional partial differential equations with Barron data
• The Barron Space and the Flow-induced Function Spaces for Neural Network Models
• Embedding Inequalities for Barron-type Spaces
• Approximation of Functionals by Neural Network without Curse of Dimensionality