Learning Two Layer Rectified Neural Networks in Polynomial Time

Abstract

Consider the following fundamental learning problem: given input examples $x\in {\mathbb{R}}^d$ and their vector-valued labels, as defined by an underlying generative neural network, recover the weight matrices of this network. We consider two-layer networks, mapping ${\mathbb{R}}^d$ to ${\mathbb{R}}^m$, with $k$ non-linear activation units $f(\cdot )$, where $f(x)=\max \{x,0\}$ is the ReLU. Such a network is specified by two weight matrices, ${\mathbf{U}}^{\ast }\in {\mathbb{R}}^{m\times k},{\mathbf{V}}^{\ast }\in {\mathbb{R}}^{k\times d}$, such that the label of an example $x\in {\mathbb{R}}^d$ is given by ${\mathbf{U}}^{\ast }f({\mathbf{V}}^{\ast }x)$, where $f(\cdot )$ is applied coordinate-wise. Given $n$ samples as a matrix $\mathbf{X}\in {\mathbb{R}}^{d\times n}$ and the (possibly noisy) labels ${\mathbf{U}}^{\ast }f({\mathbf{V}}^{\ast }\mathbf{X})+\mathbf{E}$ of the network on these samples, where $\mathbf{E}$ is a noise matrix, our goal is to recover the weight matrices ${\mathbf{U}}^{\ast }$ and ${\mathbf{V}}^{\ast }$. In this work, we develop algorithms and hardness results under varying assumptions on the input and noise. Although the problem is NP-hard even for $k=2$, by assuming Gaussian marginals over the input $\mathbf{X}$ we are able to develop polynomial time algorithms for the approximate recovery of ${\mathbf{U}}^{\ast }$ and ${\mathbf{V}}^{\ast }$. Perhaps surprisingly, in the noiseless case our algorithms recover ${\mathbf{U}}^{\ast },{\mathbf{V}}^{\ast }$ exactly, i.e., with no error. To the best of the our knowledge, this is the first algorithm to accomplish exact recovery. For the noisy case, we give the first polynomial time algorithm that approximately recovers the weights in the presence of mean-zero noise $\mathbf{E}$. Our algorithms generalize to a larger class of rectified activation functions, $f(x)=0$ when $x\le 0$, and $f(x)>0$ otherwise.

References in corpus (10)

• Sparsity and Incoherence in Compressive Sampling
• Recovery Guarantees for One-hidden-layer Neural Networks
• Learning One-hidden-layer Neural Networks with Landscape Design
• Reliably Learning the ReLU in Polynomial Time
• Learning One Convolutional Layer with Overlapping Patches
• Decomposing Overcomplete 3rd Order Tensors using Sum-of-Squares Algorithms
• Learning Neural Networks with Two Nonlinear Layers in Polynomial Time
• On the Connection Between Learning Two-Layers Neural Networks and Tensor Decomposition
• Complexity of Training ReLU Neural Network
• Provable Methods for Training Neural Networks with Sparse Connectivity

Cited by in corpus (7)

• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• What Can ResNet Learn Efficiently, Going Beyond Kernels?
• The Computational Complexity of ReLU Network Training Parameterized by Data Dimensionality
• Training a Two Layer ReLU Network Analytically
• Learning Polynomials of Few Relevant Dimensions
• A Deep Conditioning Treatment of Neural Networks