Mitigating deep double descent by concatenating inputs

Abstract

The double descent curve is one of the most intriguing properties of deep neural networks. It contrasts the classical bias-variance curve with the behavior of modern neural networks, occurring where the number of samples nears the number of parameters. In this work, we explore the connection between the double descent phenomena and the number of samples in the deep neural network setting. In particular, we propose a construction which augments the existing dataset by artificially increasing the number of samples. This construction empirically mitigates the double descent curve in this setting. We reproduce existing work on deep double descent, and observe a smooth descent into the overparameterized region for our construction. This occurs both with respect to the model size, and with respect to the number epochs.

References in corpus (10)

• Wide Residual Networks
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Benign Overfitting in Linear Regression
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Rethinking Bias-Variance Trade-off for Generalization of Neural Networks
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• More Data Can Hurt for Linear Regression: Sample-wise Double Descent
• Understanding overfitting peaks in generalization error: Analytical risk curves for $l_2$ and $l_1$ penalized interpolation
• A New Look at an Old Problem: A Universal Learning Approach to Linear Regression
• Multiple Descent: Design Your Own Generalization Curve