102 citations · 157 across the 6 of their papers we have counts for
5 papers · 1 filter
Ridgeless Interpolation with Shallow ReLU Networks in is Nearest Neighbor Curvature Extrapolation and Provably Generalizes on Lipschitz Functions
Boris Hanin
We prove a precise geometric description of all one layer ReLU networks with a single linear unit and input/output dimensions equal to one that interpolate a given dataset…
Deep ReLU Networks Preserve Expected Length
Boris Hanin, Ryan Jeong, David Rolnick
Assessing the complexity of functions computed by a neural network helps us understand how the network will learn and generalize. One natural measure of complexity is how the netwo…
Deep ReLU Networks Have Surprisingly Few Activation Patterns
Boris Hanin, David Rolnick
The success of deep networks has been attributed in part to their expressivity: per parameter, deep networks can approximate a richer class of functions than shallow networks. In R…
Complexity of Linear Regions in Deep Networks
Boris Hanin, David Rolnick
It is well-known that the expressivity of a neural network depends on its architecture, with deeper networks expressing more complex functions. In the case of networks that compute…
How to Start Training: The Effect of Initialization and Architecture
Boris Hanin, David Rolnick
We identify and study two common failure modes for early training in deep ReLU nets. For each we give a rigorous proof of when it occurs and how to avoid it, for fully connected an…