Critical Learning Periods in Federated Learning

Abstract

Federated learning (FL) is a popular technique to train machine learning (ML) models with decentralized data. Extensive works have studied the performance of the global model; however, it is still unclear how the training process affects the final test accuracy. Exacerbating this problem is the fact that FL executions differ significantly from traditional ML with heterogeneous data characteristics across clients, involving more hyperparameters. In this work, we show that the final test accuracy of FL is dramatically affected by the early phase of the training process, i.e., FL exhibits critical learning periods, in which small gradient errors can have irrecoverable impact on the final test accuracy. To further explain this phenomenon, we generalize the trace of the Fisher Information Matrix (FIM) to FL and define a new notion called FedFIM, a quantity reflecting the local curvature of each clients from the beginning of the training in FL. Our findings suggest that the {\em initial learning phase} plays a critical role in understanding the FL performance. This is in contrast to many existing works which generally do not connect the final accuracy of FL to the early phase training. Finally, seizing critical learning periods in FL is of independent interest and could be useful for other problems such as the choices of hyperparameters such as the number of client selected per round, batch size, and more, so as to improve the performance of FL training and testing.

References in corpus (14)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Differentially Private Federated Learning: A Client Level Perspective
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• New insights and perspectives on the natural gradient method
• Three Factors Influencing Minima in SGD
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• Distributed Mean Estimation with Limited Communication
• Federated Learning with Matched Averaging
• On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
• Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration
• Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
• Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization
• Straggler-Resilient Distributed Machine Learning with Dynamic Backup Workers