FedMAX: Mitigating Activation Divergence for Accurate and Communication-Efficient Federated Learning

Abstract

In this paper, we identify a new phenomenon called activation-divergence which occurs in Federated Learning (FL) due to data heterogeneity (i.e., data being non-IID) across multiple users. Specifically, we argue that the activation vectors in FL can diverge, even if subsets of users share a few common classes with data residing on different devices. To address the activation-divergence issue, we introduce a prior based on the principle of maximum entropy; this prior assumes minimal information about the per-device activation vectors and aims at making the activation vectors of same classes as similar as possible across multiple devices. Our results show that, for both IID and non-IID settings, our proposed approach results in better accuracy (due to the significantly more similar activation vectors across multiple devices), and is more communication-efficient than state-of-the-art approaches in FL. Finally, we illustrate the effectiveness of our approach on a few common benchmarks and two large medical datasets.

References in corpus (8)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• Federated Optimization:Distributed Optimization Beyond the Datacenter
• Federated Optimization in Heterogeneous Networks
• Communication Efficient Distributed Optimization using an Approximate Newton-type Method
• LEAF: A Benchmark for Federated Settings
• Federated Learning with Bayesian Differential Privacy

Cited by in corpus (6)

• Robust Federated Learning with Noisy Labels
• FedCluster: Boosting the Convergence of Federated Learning via Cluster-Cycling
• Communication Optimization in Large Scale Federated Learning using Autoencoder Compressed Weight Updates
• New Directions in Distributed Deep Learning: Bringing the Network at Forefront of IoT Design
• Demystifying the Effects of Non-Independence in Federated Learning
• Weight Divergence Driven Divide-and-Conquer Approach for Optimal Federated Learning from non-IID Data