55 citations · 81 across the 5 of their papers we have counts for
4 papers · 1 filter
Deep ReLU Networks Preserve Expected Length
Boris Hanin, Ryan Jeong, David Rolnick
Assessing the complexity of functions computed by a neural network helps us understand how the network will learn and generalize. One natural measure of complexity is how the netwo…
Deep ReLU Networks Have Surprisingly Few Activation Patterns
Boris Hanin, David Rolnick
The success of deep networks has been attributed in part to their expressivity: per parameter, deep networks can approximate a richer class of functions than shallow networks. In R…
Complexity of Linear Regions in Deep Networks
Boris Hanin, David Rolnick
It is well-known that the expressivity of a neural network depends on its architecture, with deeper networks expressing more complex functions. In the case of networks that compute…
How to Start Training: The Effect of Initialization and Architecture
Boris Hanin, David Rolnick
We identify and study two common failure modes for early training in deep ReLU nets. For each we give a rigorous proof of when it occurs and how to avoid it, for fully connected an…