Communication-Efficient Robust Federated Learning with Noisy Labels

Abstract

Federated learning (FL) is a promising privacy-preserving machine learning paradigm over distributed located data. In FL, the data is kept locally by each user. This protects the user privacy, but also makes the server difficult to verify data quality, especially if the data are correctly labeled. Training with corrupted labels is harmful to the federated learning task; however, little attention has been paid to FL in the case of label noise. In this paper, we focus on this problem and propose a learning-based reweighting approach to mitigate the effect of noisy labels in FL. More precisely, we tuned a weight for each training sample such that the learned model has optimal generalization performance over a validation set. More formally, the process can be formulated as a Federated Bilevel Optimization problem. Bilevel optimization problem is a type of optimization problem with two levels of entangled problems. The non-distributed bilevel problems have witnessed notable progress recently with new efficient algorithms. However, solving bilevel optimization problems under the Federated Learning setting is under-investigated. We identify that the high communication cost in hypergradient evaluation is the major bottleneck. So we propose \textit{Comm-FedBiO} to solve the general Federated Bilevel Optimization problems; more specifically, we propose two communication-efficient subroutines to estimate the hypergradient. Convergence analysis of the proposed algorithms is also provided. Finally, we apply the proposed algorithms to solve the noisy label problem. Our approach has shown superior performance on several real-world datasets compared to various baselines.

arXiv commentTo appear in KDD 2022

References in corpus (29)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting
• Agnostic Federated Learning
• Ditto: Fair and Robust Federated Learning Through Personalization
• Fast Context Adaptation via Meta-Learning
• On the Convergence of Local Descent Methods in Federated Learning
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• Hyperparameter optimization with approximate gradient
• Robust Federated Learning with Noisy Labels
• Variance Reduced Local SGD with Lower Communication Complexity
• Stochastic Hyperparameter Optimization through Hypernetworks
• FetchSGD: Communication-Efficient Federated Learning with Sketching
• Approximation Methods for Bilevel Programming
• FOCUS: Dealing with Label Quality Disparity in Federated Learning
• Tighter Theory for Local SGD on Identical and Heterogeneous Data
• Reviving and Improving Recurrent Back-Propagation
• A Near-Optimal Algorithm for Stochastic Bilevel Optimization via Double-Momentum
• On the Iteration Complexity of Hypergradient Computation
• Penalty Method for Inversion-Free Deep Bilevel Optimization
• Lower Bounds and Accelerated Algorithms for Bilevel Optimization
• Improved Bilevel Model: Fast and Optimal Algorithm with Theoretical Guarantee
• Fast OSCAR and OWL Regression via Safe Screening Rules
• BiAdam: Fast Adaptive Bilevel Optimization Methods
• Enhanced Bilevel Optimization via Bregman Distance
• Local Stochastic Bilevel Optimization with Momentum-Based Variance Reduction
• Distributed Dynamic Safe Screening Algorithms for Sparse Regularization