Linear Regression with Distributed Learning: A Generalization Error Perspective
arXiv:2101.09001 · doi:10.1109/TSP.2021.3106441
Abstract
Distributed learning provides an attractive framework for scaling the learning task by sharing the computational load over multiple nodes in a network. Here, we investigate the performance of distributed learning for large-scale linear regression where the model parameters, i.e., the unknowns, are distributed over the network. We adopt a statistical learning approach. In contrast to works that focus on the performance on the training data, we focus on the generalization error, i.e., the performance on unseen data. We provide high-probability bounds on the generalization error for both isotropic and correlated Gaussian data as well as sub-gaussian data. These results reveal the dependence of the generalization performance on the partitioning of the model over the network. In particular, our results show that the generalization error of the distributed solution can be substantially higher than that of the centralized solution even when the error on the training data is at the same level for both the centralized and distributed approaches. Our numerical results illustrate the performance with both real-world image data as well as synthetic data.
References in corpus (11)
- Reconciling modern machine learning practice and the bias-variance trade-off
- Understanding deep learning requires rethinking generalization
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- Benign Overfitting in Linear Regression
- Distributing the Kalman Filter for Large-Scale Systems
- Two models of double descent for weak features
- Distributed Clustering and Learning Over Networks
- Privacy-preserving Distributed Machine Learning via Local Randomization and ADMM Perturbation
- Optimal Regularization Can Mitigate Double Descent
- Distributed Constrained Online Learning
- Coded Stochastic ADMM for Decentralized Consensus Optimization with Edge Computing