Weight Sharing is Crucial to Succesful Optimization

Abstract

Exploiting the great expressive power of Deep Neural Network architectures, relies on the ability to train them. While current theoretical work provides, mostly, results showing the hardness of this task, empirical evidence usually differs from this line, with success stories in abundance. A strong position among empirically successful architectures is captured by networks where extensive weight sharing is used, either by Convolutional or Recurrent layers. Additionally, characterizing specific aspects of different tasks, making them "harder" or "easier", is an interesting direction explored both theoretically and empirically. We consider a family of ConvNet architectures, and prove that weight sharing can be crucial, from an optimization point of view. We explore different notions of the frequency, of the target function, proving necessity of the target function having some low frequency components. This necessity is not sufficient - only with weight sharing can it be exploited, thus theoretically separating architectures using it, from others which do not. Our theoretical results are aligned with empirical experiments in an even more general setting, suggesting viability of examination of the role played by interleaving those aspects in broader families of tasks.

References in corpus (7)

• Provable Bounds for Learning Some Deep Representations
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
• Global Optimality in Tensor Factorization, Deep Learning, and Beyond
• Failures of Gradient-Based Deep Learning
• The Landscape of Empirical Risk for Non-convex Losses
• Distribution-Specific Hardness of Learning Neural Networks

Cited by in corpus (4)

• ResNet with one-neuron hidden layers is a Universal Approximator
• The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
• Learning Activation Functions: A new paradigm for understanding Neural Networks
• Computational Separation Between Convolutional and Fully-Connected Networks