Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge

Abstract

Scaling up the convolutional neural network (CNN) size (e.g., width, depth, etc.) is known to effectively improve model accuracy. However, the large model size impedes training on resource-constrained edge devices. For instance, federated learning (FL) may place undue burden on the compute capability of edge nodes, even though there is a strong practical need for FL due to its privacy and confidentiality properties. To address the resource-constrained reality of edge devices, we reformulate FL as a group knowledge transfer training algorithm, called FedGKT. FedGKT designs a variant of the alternating minimization approach to train small CNNs on edge nodes and periodically transfer their knowledge by knowledge distillation to a large server-side CNN. FedGKT consolidates several advantages into a single framework: reduced demand for edge computation, lower communication bandwidth for large CNNs, and asynchronous training, all while maintaining model accuracy comparable to FedAvg. We train CNNs designed based on ResNet-56 and ResNet-110 using three distinct datasets (CIFAR-10, CIFAR-100, and CINIC-10) and their non-I.I.D. variants. Our results show that FedGKT can obtain comparable or even slightly higher accuracy than FedAvg. More importantly, FedGKT makes edge training affordable. Compared to the edge training using FedAvg, FedGKT demands 9 to 17 times less computational power (FLOPs) on edge devices and requires 54 to 105 times fewer parameters in the edge CNN. Our source code is released at FedML (https://fedml.ai).

arXiv commentThis paper is accepted to NeurIPS 2020. We propose FedGKT, attempting to address one of the core problems of federated learning: training deep neural networks in resource-constrained edge devices

References in corpus (45)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• FitNets: Hints for Thin Deep Nets
• Do Deep Nets Really Need to be Deep?
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Towards Federated Learning at Scale: System Design
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Federated Optimization in Heterogeneous Networks
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• Knowledge Distillation by On-the-Fly Native Ensemble
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data
• LEAF: A Benchmark for Federated Settings
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• Communication Compression for Decentralized Training
• Split learning for health: Distributed deep learning without sharing raw patient data
• Large scale distributed neural network training through online distillation
• Adaptive Federated Optimization
• Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
• The Non-IID Data Quagmire of Decentralized Machine Learning
• Measuring the Algorithmic Efficiency of Neural Networks
• Distilled One-Shot Federated Learning
• Central Server Free Federated Learning over Single-sided Trust Social Networks
• An Exponential Learning Rate Schedule for Deep Learning
• Hydra: Preserving Ensemble Diversity for Model Distillation
• Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Search
• Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learning
• Distilling On-Device Intelligence at the Network Edge
• Federated Model Distillation with Noise-Free Differential Privacy
• Secure Aggregation with Heterogeneous Quantization in Federated Learning
• On Learning Rates and Schrödinger Operators
• Byzantine-Resilient Secure Federated Learning
• MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation
• Practical One-Shot Federated Learning for Cross-Silo Setting
• Edge Bias in Federated Learning and its Solution by Buffered Knowledge Distillation
• Online Knowledge Distillation with Diverse Peers
• Adaptive Distillation for Decentralized Learning from Heterogeneous Clients

Cited by in corpus (17)

• Knowledge Distillation: A Survey
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imaging
• SelfFed: Self-Supervised Federated Learning for Data Heterogeneity and Label Scarcity in Medical Images
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• TornadoAggregate: Accurate and Scalable Federated Learning via the Ring-Based Architecture
• Parameterized Knowledge Transfer for Personalized Federated Learning
• FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusion
• Edge-cloud Collaborative Learning with Federated and Centralized Features
• A Robust Federated Learning Approach for Combating Attacks Against IoT Systems Under non-IID Challenges
• CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Community
• Communication Efficiency Optimization of Federated Learning for Computing and Network Convergence of 6G Networks
• FedHome: Cloud-Edge based Personalized Federated Learning for In-Home Health Monitoring
• A Communication-efficient Local Differentially Private Algorithm in Federated Optimization
• Exploiting Features and Logits in Heterogeneous Federated Learning
• TOFU: Towards Obfuscated Federated Updates by Encoding Weight Updates into Gradients from Proxy Data
• FedOptima: Optimizing Resource Utilization in Federated Learning