Tropical Geometry of Deep Neural Networks
arXiv:1805.07091
Abstract
We establish, for the first time, connections between feedforward neural networks with ReLU activation and tropical geometry --- we show that the family of such neural networks is equivalent to the family of tropical rational maps. Among other things, we deduce that feedforward ReLU neural networks with one hidden layer can be characterized by zonotopes, which serve as building blocks for deeper networks; we relate decision boundaries of such neural networks to tropical hypersurfaces, a major object of study in tropical geometry; and we prove that linear regions of such neural networks correspond to vertices of polytopes associated with tropical rational functions. An insight from our tropical formulation is that a deeper network is exponentially more expressive than a shallow network.
arXiv comment18 pages, 6 figures
References in corpus (3)
Cited by in corpus (13)
- A Tropical Approach to Neural Networks with Piecewise Linear Activations
- Some open questions on morphological operators and representations in the deep learning era
- Algebraic solution of tropical best approximation problems
- Some Open Questions in Quiver Gauge Theory
- Improved Generalization Bounds of Group Invariant / Equivariant Deep Networks via Quotient Feature Spaces
- Spectral inequalities for nonnegative tensors and their tropical analogues
- Linear and Rational Factorization of Tropical Polynomials
- Deep Networks as Logical Circuits: Generalization and Interpretation
- Size-Noise Tradeoffs in Generative Networks
- Sparse Approximate Solutions to Max-Plus Equations with Application to Multivariate Convex Regression
- Bounding The Number of Linear Regions in Local Area for Neural Networks with ReLU Activations
- Expressive Power and Loss Surfaces of Deep Learning Models
- Interpretable Factorization for Neural Network ECG Models