ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Models

Suzanna Parkinson, Greg Ongie, Rebecca Willett

Neural networks often operate in the overparameterized regime, in which there are far more parameters than training samples, allowing the training data to be fit perfectly. That is…