Distilling On-Device Intelligence at the Network Edge

Abstract

Devices at the edge of wireless networks are the last mile data sources for machine learning (ML). As opposed to traditional ready-made public datasets, these user-generated private datasets reflect the freshest local environments in real time. They are thus indispensable for enabling mission-critical intelligent systems, ranging from fog radio access networks (RANs) to driverless cars and e-Health wearables. This article focuses on how to distill high-quality on-device ML models using fog computing, from such user-generated private data dispersed across wirelessly connected devices. To this end, we introduce communication-efficient and privacy-preserving distributed ML frameworks, termed fog ML (FML), wherein on-device ML models are trained by exchanging model parameters, model outputs, and surrogate data. We then present advanced FML frameworks addressing wireless RAN characteristics, limited on-device resources, and imbalanced data distributions. Our study suggests that the full potential of FML can be reached by co-designing communication and distributed ML operations while accounting for heterogeneous hardware specifications, data characteristics, and user requirements.

7 pages, 6 figures; This work has been submitted to the IEEE for possible publication

References in corpus (6)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• Large scale distributed neural network training through online distillation
• No Peek: A Survey of private distributed deep learning
• Multi-hop Federated Private Data Augmentation with Sample Compression
• Robust Coreset Construction for Distributed Machine Learning

Cited by in corpus (6)

• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Robust Blockchained Federated Learning with Model Validation and Proof-of-Stake Inspired Consensus
• Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space
• Federated Learning for Energy Constrained IoT devices: A systematic mapping study
• Robustness and Diversity Seeking Data-Free Knowledge Distillation
• Mix2FLD: Downlink Federated Learning After Uplink Federated Distillation With Two-Way Mixup