A note on Linear Bottleneck networks and their Transition to Multilinearity

Libin Zhu, Parthe Pandit, Mikhail Belkin

Randomly initialized wide neural networks transition to linear functions of weights as the width grows, in a ball of radius $O(1)$ around initialization. A necessary condition for…