AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learning

Abstract

Federated learning enables a cluster of decentralized mobile devices at the edge to collaboratively train a shared machine learning model, while keeping all the raw training samples on device. This decentralized training approach is demonstrated as a practical solution to mitigate the risk of privacy leakage. However, enabling efficient FL deployment at the edge is challenging because of non-IID training data distribution, wide system heterogeneity and stochastic-varying runtime effects in the field. This paper jointly optimizes time-to-convergence and energy efficiency of state-of-the-art FL use cases by taking into account the stochastic nature of edge execution. We propose AutoFL by tailor-designing a reinforcement learning algorithm that learns and determines which K participant devices and per-device execution targets for each FL model aggregation round in the presence of stochastic runtime variance, system and data heterogeneity. By considering the unique characteristics of FL edge deployment judiciously, AutoFL achieves 3.6 times faster model convergence time and 4.7 and 5.2 times higher energy efficiency for local clients and globally over the cluster of K participants, respectively.

References in corpus (22)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• Federated Learning for Mobile Keyboard Prediction
• Towards Federated Learning at Scale: System Design
• Differentially Private Federated Learning: A Client Level Perspective
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Agnostic Federated Learning
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• Federated Learning on Non-IID Data Silos: An Experimental Study
• LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets
• ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
• Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems
• A first look into the carbon footprint of federated learning
• Synergy: A HW/SW Framework for High Throughput CNNs on Embedded Heterogeneous SoC
• Adapting multi-armed bandits policies to contextual bandits scenarios
• Asynchronous Federated Learning with Reduced Number of Rounds and with Differential Privacy from Less Aggregated Gaussian Noise
• Lottery Hypothesis based Unsupervised Pre-training for Model Compression in Federated Learning
• Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training