Recovering the Lowest Layer of Deep Networks with High Threshold Activations

Abstract

Giving provable guarantees for learning neural networks is a core challenge of machine learning theory. Most prior work gives parameter recovery guarantees for one hidden layer networks, however, the networks used in practice have multiple non-linear layers. In this work, we show how we can strengthen such results to deeper networks -- we address the problem of uncovering the lowest layer in a deep neural network under the assumption that the lowest layer uses a high threshold before applying the activation, the upper network can be modeled as a well-behaved polynomial and the input distribution is Gaussian.

References in corpus (6)

• Recovery Guarantees for One-hidden-layer Neural Networks
• Learning One-hidden-layer Neural Networks with Landscape Design
• When is a Convolutional Filter Easy To Learn?
• Reliably Learning the ReLU in Polynomial Time
• Learning Non-overlapping Convolutional Neural Networks with Multiple Kernels
• Learning Neural Networks with Two Nonlinear Layers in Polynomial Time