On the Learnability of Deep Random Networks

Abstract

In this paper we study the learnability of deep random networks from both theoretical and practical points of view. On the theoretical front, we show that the learnability of random deep networks with sign activation drops exponentially with its depth. On the practical front, we find that the learnability drops sharply with depth even with the state-of-the-art training methods, suggesting that our stylized theoretical results are closer to reality.

References in corpus (8)

• Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
• Efficient approaches for escaping higher order saddle points in non-convex optimization
• Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs
• When is a Convolutional Filter Easy To Learn?
• Reliably Learning the ReLU in Polynomial Time
• Learning Non-overlapping Convolutional Neural Networks with Multiple Kernels
• Learning Neural Networks with Two Nonlinear Layers in Polynomial Time
• Learning Halfspaces and Neural Networks with Random Initialization

Cited by in corpus (5)

• High Accuracy and High Fidelity Extraction of Neural Networks
• Hardness of Learning Neural Networks with Natural Weights
• A Deep Conditioning Treatment of Neural Networks
• From Local Pseudorandom Generators to Hardness of Learning
• Learning Boolean Circuits with Neural Networks