Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data

Abstract

On-device machine learning (ML) enables the training process to exploit a massive amount of user-generated private data samples. To enjoy this benefit, inter-device communication overhead should be minimized. With this end, we propose federated distillation (FD), a distributed model training algorithm whose communication payload size is much smaller than a benchmark scheme, federated learning (FL), particularly when the model size is large. Moreover, user-generated data samples are likely to become non-IID across devices, which commonly degrades the performance compared to the case with an IID dataset. To cope with this, we propose federated augmentation (FAug), where each device collectively trains a generative model, and thereby augments its local data towards yielding an IID dataset. Empirical studies demonstrate that FD with FAug yields around 26x less communication overhead while achieving 95-98% test accuracy compared to FL.

presented at the 32nd Conference on Neural Information Processing Systems (NIPS 2018), 2nd Workshop on Machine Learning on the Phone and other Consumer Devices (MLPCD 2), Montréal, Canada

References in corpus (2)

• Distilling the Knowledge in a Neural Network
• Differentially Private Federated Learning: A Client Level Perspective

Cited by in corpus (83)

• Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Federated Learning with Cooperating Devices: A Consensus Approach for Massive IoT Networks
• Federated Optimization in Heterogeneous Networks
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Data
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Analysis and Optimal Edge Assignment For Hierarchical Federated Learning on Non-IID Data
• Variational Federated Multi-Task Learning
• A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
• Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer
• FedSAE: A Novel Self-Adaptive Federated Learning Framework in Heterogeneous Systems
• Federated Graph Classification over Non-IID Graphs
• Decentralized Deep Learning for Multi-Access Edge Computing: A Survey on Communication Efficiency and Trustworthiness
• XOR Mixup: Privacy-Preserving Data Augmentation for One-Shot Federated Learning
• FedICT: Federated Multi-task Distillation for Multi-access Edge Computing
• FedCluster: Boosting the Convergence of Federated Learning via Cluster-Cycling
• Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating
• GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning
• Model Pruning Enables Localized and Efficient Federated Learning for Yield Forecasting and Data Sharing
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data
• Time-sensitive Learning for Heterogeneous Federated Edge Intelligence
• Auto-FedAvg: Learnable Federated Averaging for Multi-Institutional Medical Image Segmentation
• Faster On-Device Training Using New Federated Momentum Algorithm
• Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs
• Distilling On-Device Intelligence at the Network Edge
• Communication-Efficient Local Decentralized SGD Methods
• Distributed Learning in Wireless Networks: Recent Progress and Future Challenges
• Decentralized federated learning of deep neural networks on non-iid data
• Personalized Federated Learning for Intelligent IoT Applications: A Cloud-Edge based Framework
• TornadoAggregate: Accurate and Scalable Federated Learning via the Ring-Based Architecture
• Privacy Sensitive Speech Analysis Using Federated Learning to Assess Depression
• Decentralized Learning of Generative Adversarial Networks from Non-iid Data
• Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space
• Parameterized Knowledge Transfer for Personalized Federated Learning
• GFL: A Decentralized Federated Learning Framework Based On Blockchain
• Multi-hop Federated Private Data Augmentation with Sample Compression
• FedEval: A Holistic Evaluation Framework for Federated Learning
• Patient Clustering Improves Efficiency of Federated Machine Learning to predict mortality and hospital stay time using distributed Electronic Medical Records
• Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration
• Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models
• CDKT-FL: Cross-Device Knowledge Transfer using Proxy Dataset in Federated Learning
• Federated Learning for Energy Constrained IoT devices: A systematic mapping study
• Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning
• Specialized federated learning using a mixture of experts
• Accurate and Fast Federated Learning via IID and Communication-Aware Grouping
• Communication-Efficient Federated Distillation
• Homogeneous Learning: Self-Attention Decentralized Deep Learning
• Communication Efficient Distributed Learning with Censored, Quantized, and Generalized Group ADMM
• Gradient Scheduling with Global Momentum for Non-IID Data Distributed Asynchronous Training
• Federated Learning for Commercial Image Sources
• From Federated Learning to Federated Neural Architecture Search: A Survey
• Heterogeneous Data-Aware Federated Learning
• Federated Learning on Non-iid Data via Local and Global Distillation
• Aggregate or Not? Exploring Where to Privatize in DNN Based Federated Learning Under Different Non-IID Scenes
• Wireless for Machine Learning
• Distributed Heteromodal Split Learning for Vision Aided mmWave Received Power Prediction
• Federated Action Recognition on Heterogeneous Embedded Devices
• Resource-Constrained Federated Learning with Heterogeneous Labels and Models
• Efficient Ring-topology Decentralized Federated Learning with Deep Generative Models for Industrial Artificial Intelligent
• An Energy Consumption Model for Electrical Vehicle Networks via Extended Federated-learning
• Federated Learning for Open Banking
• Integrating LEO Satellites and Multi-UAV Reinforcement Learning for Hybrid FSO/RF Non-Terrestrial Networks
• Towards Efficient Scheduling of Federated Mobile Devices under Computational and Statistical Heterogeneity
• Performance Analysis and Characterization of Training Deep Learning Models on Mobile Devices
• Adversarial Robustness through Bias Variance Decomposition: A New Perspective for Federated Learning
• Adaptive Distillation for Decentralized Learning from Heterogeneous Clients
• The Disruptions of 5G on Data-driven Technologies and Applications
• Collate: Collaborative Neural Network Learning for Latency-Critical Edge Systems
• Cooperative Learning via Federated Distillation over Fading Channels
• Asynchronous Federated Learning for Sensor Data with Concept Drift
• Reward-Based 1-bit Compressed Federated Distillation on Blockchain
• Dynamic Attention-based Communication-Efficient Federated Learning
• Preliminary Steps Towards Federated Sentiment Classification
• More Industry-friendly: Federated Learning with High Efficient Design
• Distributed U-net model and Image Segmentation for Lung Cancer Detection
• Edge Learning with Timeliness Constraints: Challenges and Solutions
• Mix2FLD: Downlink Federated Learning After Uplink Federated Distillation With Two-Way Mixup
• Security and Privacy Issues and Solutions in Federated Learning for Digital Healthcare
• Adaptive Subcarrier, Parameter, and Power Allocation for Partitioned Edge Learning Over Broadband Channels
• Distantly Supervised Relation Extraction in Federated Settings
• Efficient Federated Learning for AIoT Applications Using Knowledge Distillation