An Embedding of ReLU Networks and an Analysis of their Identifiability

Abstract

Neural networks with the Rectified Linear Unit (ReLU) nonlinearity are described by a vector of parameters $\theta$, and realized as a piecewise linear continuous function $R_{\theta }:x\in {\mathbb{R}}^d\mapsto R_{\theta }(x)\in {\mathbb{R}}^k$. Natural scalings and permutations operations on the parameters $\theta$ leave the realization unchanged, leading to equivalence classes of parameters that yield the same realization. These considerations in turn lead to the notion of identifiability -- the ability to recover (the equivalence class of) $\theta$ from the sole knowledge of its realization $R_{\theta }$. The overall objective of this paper is to introduce an embedding for ReLU neural networks of any depth, $\Phi (\theta )$, that is invariant to scalings and that provides a locally linear parameterization of the realization of the network. Leveraging these two key properties, we derive some conditions under which a deep ReLU network is indeed locally identifiable from the knowledge of the realization on a finite set of samples $x_i\in {\mathbb{R}}^d$. We study the shallow case in more depth, establishing necessary and sufficient conditions for the network to be identifiable from a bounded subset $\mathcal{X}\subseteq {\mathbb{R}}^d$.

arXiv commentConstructive Approximation camera-ready

References in corpus (10)

• On the Number of Linear Regions of Deep Neural Networks
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• On the number of response regions of deep feed forward networks with piece-wise linear activations
• Deep ReLU Networks Have Surprisingly Few Activation Patterns
• Nonlinear Approximation and (Deep) ReLU Networks
• Norm-Based Capacity Control in Neural Networks
• Reverse-Engineering Deep ReLU Networks
• Positively Scale-Invariant Flatness of ReLU Neural Networks
• Equi-normalization of Neural Networks
• Robust and Resource Efficient Identification of Two Hidden Layer Neural Networks

Cited by in corpus (1)

• Functional dimension of feedforward ReLU neural networks