Data splitting improves statistical performance in overparametrized regimes

Abstract

While large training datasets generally offer improvement in model performance, the training process becomes computationally expensive and time consuming. Distributed learning is a common strategy to reduce the overall training time by exploiting multiple computing devices. Recently, it has been observed in the single machine setting that overparametrization is essential for benign overfitting in ridgeless regression in Hilbert spaces. We show that in this regime, data splitting has a regularizing effect, hence improving statistical performance and computational complexity at the same time. We further provide a unified framework that allows to analyze both the finite and infinite dimensional setting. We numerically demonstrate the effect of different model parameters.

References in corpus (6)

• Divide and Conquer Kernel Ridge Regression: A Distributed Algorithm with Minimax Optimal Rates
• Optimal Rates for Spectral Algorithms with Least-Squares Regression over Hilbert Spaces
• A review of distributed statistical inference
• Asymptotics of Ridge (less) Regression under General Source Condition
• Ivanov-Regularised Least-Squares Estimators over Large RKHSs and Their Interpolation Spaces
• On the Universality of the Double Descent Peak in Ridgeless Regression