On the Expected Complexity of Maxout Networks

Abstract

Learning with neural networks relies on the complexity of the representable functions, but more importantly, the particular assignment of typical parameters to functions of different complexity. Taking the number of activation regions as a complexity measure, recent works have shown that the practical complexity of deep ReLU networks is often far from the theoretical maximum. In this work, we show that this phenomenon also occurs in networks with maxout (multi-argument) activation functions and when considering the decision boundaries in classification tasks. We also show that the parameter space has a multitude of full-dimensional regions with widely different complexity, and obtain nontrivial lower bounds on the expected complexity. Finally, we investigate different parameter initialization procedures and show that they can increase the speed of convergence in training.

arXiv commentPublished at NeurIPS 2021, 47 pages, 18 figures

References in corpus (16)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• On the Number of Linear Regions of Deep Neural Networks
• Representation Benefits of Deep Feedforward Networks
• Deep ReLU Networks Have Surprisingly Few Activation Patterns
• Provable Robustness of ReLU networks via Maximization of Linear Regions
• Complexity of Linear Regions in Deep Networks
• Products of Many Large Random Matrices and Gradients in Deep Neural Networks
• Weisfeiler and Lehman Go Topological: Message Passing Simplicial Networks
• Empirical Studies on the Properties of Linear Regions in Deep Neural Networks
• On the Number of Linear Regions of Convolutional Neural Networks
• A Tropical Approach to Neural Networks with Piecewise Linear Activations
• Deep ReLU Networks Preserve Expected Length
• A Sober Look at Neural Network Initializations
• Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sums
• Lefschetz and Lower Bound theorems for Minkowski sums
• Using activation histograms to bound the number of affine regions in ReLU feed-forward neural networks