Data-Free Evaluation of User Contributions in Federated Learning

Abstract

Federated learning (FL) trains a machine learning model on mobile devices in a distributed manner using each device's private data and computing resources. A critical issues is to evaluate individual users' contributions so that (1) users' effort in model training can be compensated with proper incentives and (2) malicious and low-quality users can be detected and removed. The state-of-the-art solutions require a representative test dataset for the evaluation purpose, but such a dataset is often unavailable and hard to synthesize. In this paper, we propose a method called Pairwise Correlated Agreement (PCA) based on the idea of peer prediction to evaluate user contribution in FL without a test dataset. PCA achieves this using the statistical correlation of the model parameters uploaded by users. We then apply PCA to designing (1) a new federated learning algorithm called Fed-PCA, and (2) a new incentive mechanism that guarantees truthfulness. We evaluate the performance of PCA and Fed-PCA using the MNIST dataset and a large industrial product recommendation dataset. The results demonstrate that our Fed-PCA outperforms the canonical FedAvg algorithm and other baseline methods in accuracy, and at the same time, PCA effectively incentivizes users to behave truthfully.

arXiv commentaccepted by WiOpt 2021

References in corpus (15)

• Deep Learning with Differential Privacy
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• Federated Learning for Mobile Keyboard Prediction
• Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• FMore: An Incentive Scheme of Multi-dimensional Auction for Federated Learning in MEC
• Byzantine-Robust Federated Machine Learning through Adaptive Model Averaging
• Free-riders in Federated Learning: Attacks and Defenses
• Deep Learning with Gaussian Differential Privacy
• FedCoin: A Peer-to-Peer Payment System for Federated Learning
• Incentives for Federated Learning: a Hypothesis Elicitation Approach
• An Exploratory Analysis on Users' Contributions in Federated Learning
• FedServing: A Federated Prediction Serving Framework Based on Incentive Mechanism