Is Deeper Better only when Shallow is Good?

Abstract

Understanding the power of depth in feed-forward neural networks is an ongoing challenge in the field of deep learning theory. While current works account for the importance of depth for the expressive power of neural-networks, it remains an open question whether these benefits are exploited during a gradient-based optimization process. In this work we explore the relation between expressivity properties of deep networks and the ability to train them efficiently using gradient-based algorithms. We give a depth separation argument for distributions with fractal structure, showing that they can be expressed efficiently by deep networks, but not with shallow ones. These distributions have a natural coarse-to-fine structure, and we show that the balance between the coarse and fine details has a crucial effect on whether the optimization process is likely to succeed. We prove that when the distribution is concentrated on the fine details, gradient-based algorithms are likely to fail. Using this result we prove that, at least in some distributions, the success of learning deep networks depends on whether the distribution can be well approximated by shallower networks, and we conjecture that this property holds in general.

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• On the Number of Linear Regions of Deep Neural Networks
• Representation Benefits of Deep Feedforward Networks
• On the number of response regions of deep feed forward networks with piece-wise linear activations
• Benefits of depth in neural networks
• On the Expressive Power of Deep Neural Networks
• Failures of Gradient-Based Deep Learning
• Bounding and Counting Linear Regions of Deep Neural Networks

Cited by in corpus (8)

• Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks
• Depth-Width Trade-offs for ReLU Networks via Sharkovsky's Theorem
• Depth-Width Trade-offs for Neural Networks via Topological Entropy
• Better Depth-Width Trade-offs for Neural Networks through the lens of Dynamical Systems
• Learning Boolean Circuits with Neural Networks
• The staircase property: How hierarchical structure can guide deep learning
• Expressivity of Neural Networks via Chaotic Itineraries beyond Sharkovsky's Theorem
• A simple geometric proof for the benefit of depth in ReLU networks